Cross-modal image-text matching method and system based on ranking consistency and electronic device

Abstract

The application provides a cross-modal image-text matching method and system based on ranking consistency, and an electronic device, and belongs to the technical field of deep learning. The method comprises the following steps: obtaining a training set containing multiple sample pairs; obtaining a training set containing multiple sample pairs composed of image samples and text samples; evaluating the semantic matching confidence of each sample pair based on the bidirectional cross-modal retrieval result between the image and the text of each sample pair; selecting a candidate set from the training set according to the semantic matching confidence; calculating the first and second ranking similarities of each sample pair in the candidate set; calculating the final weight of the sample pair based on the first and second ranking similarities; training the sample pair in the candidate set according to the final weight, updating the parameters of the cross-modal alignment model; and inputting the data to be matched into the updated model to obtain cross-modal features, and obtaining the corresponding image-text matching result according to the similarity between the cross-modal features. The application improves the precision of image-text matching.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, specifically to a cross-modal image-text matching method, system, and electronic device based on ranking consistency. Background Technology

[0002] Cross-modal alignment learning (such as image-text alignment) aims to map data from different modalities to a unified semantic space, bringing semantically related data closer together. Its success heavily relies on large-scale, correctly aligned multimodal datasets. However, manual annotation is costly and time-consuming, making it difficult to meet the needs of large-scale model training. While multimodal pairs crawled from the internet (such as CLIP training data) offer better scalability, they inevitably introduce mismatched or only partially related multimodal data pairs; for example, an image and its corresponding alt text may be completely unrelated or only weakly related. This phenomenon of incorrectly labeling semantically mismatched or only partially related multimodal data pairs as positive samples is called "noisy correspondence." Training cross-modal alignment models on data containing noisy correspondences forces the model to fit spurious relevance, severely degrading the model's cross-modal retrieval performance. Therefore, noisy correspondence learning has become an important research direction in the field of cross-modal learning.

[0003] To address the aforementioned issues, various approaches have been proposed, including OpenAI's CLIP (Contrastive Language-Image Pre-Training) model and its training methods, Google's cross-modal learning framework, Microsoft's multimodal understanding system, and various noise correspondence learning methods from academia such as NCR (Noisy Correspondence Rectification), NPC (Noisy Pseudo-Correction), GLP (Global-Local Prompt), BiCro (Bidirectional Cross-modal Consistency), and DECL (Debiased Contrastive Learning). NCR uses the DivideMix sample selection strategy to train samples into clean and noisy subsets, converting hard labels into soft labels using the similarity predicted by the model for training. NPC estimates soft labels based on the model output by analyzing the changes in model predictions before and after gradient updates at clean anchor points. GLP utilizes global and local cue information to assist in noise correspondence learning. The above methods all attempt to solve the problem from different angles. In essence, they all evaluate the semantic matching degree of each multimodal data pair based on the image-text similarity score predicted by the model itself, and reweight the training samples accordingly.

[0004] However, the above methods share the following common drawbacks: In noisy training environments, deep neural networks gradually overfit to noisy labels during training, causing the similarity scores output by the model to be misleading and distorted. These distorted similarity scores cannot accurately reflect the semantic matching degree of the samples. Specifically, firstly, the similarity scores predicted by the model itself are unreliable. Existing technologies use these unreliable predictions as the basis for sample reweighting, leading to interference from erroneous supervisory signals during training. Secondly, existing technologies struggle to effectively distinguish between fully aligned sample pairs and only partially related sample pairs, often assigning inappropriately high similarity scores to partially related sample pairs, causing these noisy samples to receive undue high weights and further interfering with the model's learning process. Thirdly, existing technologies only focus on the similarity scores of individual sample pairs, failing to utilize the relative positional relationships of data within the global feature space manifold structure, lacking global structure awareness, resulting in a lack of global consistency in the decision to reweight samples. Summary of the Invention

[0005] In view of this, it is necessary to provide a cross-modal image-text matching method, system and electronic device based on ranking consistency, in order to solve the technical problem in the prior art that the reliance on the model's own similarity prediction leads to inaccurate sample evaluation in noisy environments, which in turn leads to inaccurate image-text matching.

[0006] To address the aforementioned technical problems, in a first aspect, the present invention provides a cross-modal image-text matching method based on order consistency, comprising: Obtain a training set, which contains multiple sample pairs consisting of image samples and text samples; Based on the bidirectional cross-modal retrieval results of each sample pair between images and text, the semantic matching confidence of each sample pair is evaluated. A candidate set is selected from the training set based on the semantic matching confidence. Calculate a first ranking similarity for each sample pair in the candidate set in the text retrieval direction, and calculate a second ranking similarity for each sample pair in the candidate set in the image retrieval direction; Based on the first ranking similarity and the second ranking similarity, the final weight of the sample pair is calculated, and the final weight is positively correlated with the semantic matching degree of the sample pair; The parameters of the cross-modal alignment model are updated by training the sample pairs in the candidate set based on the final weights. The data to be matched is input into the updated model to obtain cross-modal features, and the image-text matching result corresponding to the data to be matched is obtained based on the similarity between the cross-modal features.

[0007] In one possible implementation, evaluating the semantic matching confidence of each sample pair based on the bidirectional cross-modal retrieval results between the images and text includes: Extract the image feature vector of the image sample and extract the text feature vector of the text sample; Calculate the similarity between the image feature vector and the text feature vector; The first posterior probability of the image sample retrieving the corresponding text sample and the second posterior probability of the text sample retrieving the corresponding image sample are calculated based on the similarity. The first posterior probability and the second posterior probability constitute the bidirectional cross-modal retrieval result. The first contrast loss is obtained based on the first posterior probability, and the second contrast loss is obtained based on the second posterior probability. The semantic matching confidence of each sample pair is evaluated based on the first contrast loss and the second contrast loss.

[0008] In one possible implementation, the step of selecting a candidate set from the training set based on the semantic matching confidence includes: The Gaussian mixture model is used to fit the first distribution of the first contrastive loss and the second distribution of the second contrastive loss for each sample pair, respectively, wherein the Gaussian mixture model includes a low loss component and a high loss component; For each sample pair, calculate the first probability that the first contrastive loss of the sample pair belongs to the low loss component and the second probability that the second contrastive loss belongs to the low loss component. Sample pairs with both the first probability and the second probability greater than the preset threshold are selected and added to the candidate set.

[0009] In one possible implementation, calculating the first ranking similarity of each sample pair in the candidate set in the text retrieval direction includes: Based on the image samples, a search is performed in the text library to obtain a first actual sorted list; Based on the text sample, a first ideal sorted list is obtained by searching the text database. The first sorting similarity is calculated based on the first actual sorting list and the first ideal sorting list.

[0010] In one possible implementation, calculating the first sorting similarity based on the first actual sorting list and the first ideal sorting list includes: Obtain the first intramodal similarity between the top K texts in the first actual sorting list and the text sample; calculate the corresponding first gain value based on the first intramodal similarity corresponding to each ranking position; and calculate the first actual loss cumulative gain based on the first gain value of each ranking position and the position loss factor. Obtain the second intramodal similarity between the top K texts in the first ideal ranking list and the text sample; calculate the corresponding second gain value based on the second intramodal similarity corresponding to each ranking position; and calculate the first ideal loss cumulative gain based on the second gain value of each ranking position and the position loss factor. The first ranking similarity is calculated based on the first actual cumulative loss gain and the first ideal cumulative loss gain.

[0011] In one possible implementation, calculating the second ranking similarity of each sample pair in the candidate set in the image retrieval direction includes: Based on the text sample, a second actual sorted list is obtained by searching the image library. Based on the image samples, a second ideal sorted list is obtained by searching the image library. The second sorting similarity is calculated based on the second actual sorting list and the second ideal sorting list.

[0012] In one possible implementation, calculating the second sorting similarity based on the second actual sorting list and the second ideal sorting list includes: Obtain the intra-modal similarity between the top K images in the second actual ranking list and the image samples. Calculate the corresponding third gain value based on the intra-modal similarity of each ranking position. Calculate the second actual loss cumulative gain based on the third gain value of each ranking position and the position loss factor. Obtain the intra-modal similarity between the top K images in the second ideal ranking list and the image samples. Calculate the corresponding fourth gain value based on the intra-modal similarity of each ranking position. Calculate the second ideal loss cumulative gain based on the fourth gain value of each ranking position and the position loss factor. The second ranking similarity is calculated based on the second actual cumulative loss gain and the second ideal cumulative loss gain.

[0013] In one possible implementation, training the sample pairs in the candidate set according to the final weights and updating the parameters of the cross-modal alignment model includes: The contribution of each sample pair in the candidate set during training is determined based on the final weights. The corresponding contrast loss is weighted according to the contribution, and the parameters of the cross-modal alignment model are updated according to the weighted contrast loss.

[0014] Secondly, the present invention also provides a cross-modal image-text matching system based on ranking consistency, comprising: The data acquisition module is used to acquire a training set, which contains multiple sample pairs consisting of image samples and text samples; The retrieval evaluation module is used to evaluate the semantic matching confidence of each sample pair based on the bidirectional cross-modal retrieval results between the images and text. The data filtering module is used to filter candidate sets from the training set based on the semantic matching confidence. A similarity calculation module is used to calculate a first ranking similarity of each sample pair in the candidate set in the text retrieval direction, and to calculate a second ranking similarity of each sample pair in the candidate set in the image retrieval direction; The weight calculation module is used to calculate the final weight of the sample pair based on the first ranking similarity and the second ranking similarity, wherein the final weight is positively correlated with the semantic matching degree of the sample pair; The model update module is used to train the sample pairs in the candidate set according to the final weights and update the parameters of the cross-modal alignment model. The image-text matching module is used to input the data to be matched into the updated model to obtain cross-modal features, and obtain the image-text matching result corresponding to the data to be matched based on the similarity between the cross-modal features.

[0015] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the cross-modal image-text matching method based on order consistency described in any of the above implementations.

[0016] The beneficial effects of this invention are as follows: First, by calculating the bidirectional contrastive loss for each sample pair and using a Gaussian mixture model for coarse filtering to eliminate clearly mismatched sample pairs, this invention avoids the cross-modal alignment model learning false cross-modal relevance from obviously erroneous sample pairs, effectively reducing training interference caused by noise correspondences. Furthermore, by calculating the first and second bidirectional ranking similarities of each sample pair in the candidate set in both text retrieval and image retrieval directions to evaluate its global semantic matching degree, this invention can accurately distinguish between fully aligned sample pairs and only partially related sample pairs, avoiding interference from misleading information in partially related sample pairs, and significantly improving the retrieval accuracy and robustness of the cross-modal alignment model on noisy training data. Furthermore, by weighting the sample pairs in the candidate set based on ranking similarity and iteratively executing the above process, this invention guides the model to gradually concentrate learning resources on sample pairs with high semantic matching during training, thereby significantly reducing the retrieval performance degradation caused by noise correspondences. In practical applications of image-text matching, the model can pay more attention to the alignment degree between images and text at the global semantic level, thereby significantly improving the accuracy and quality of cross-modal image-text matching. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of an embodiment of the cross-modal image-text matching method based on order consistency provided by the present invention; Figure 2 For the present invention Figure 1A schematic diagram of an embodiment of S102; Figure 3 For the present invention Figure 1 A schematic diagram of an embodiment of S103; Figure 4 For the present invention Figure 1 A schematic diagram of an embodiment of S104; Figure 5 For the present invention Figure 4 A schematic diagram of an embodiment of S403; Figure 6 For the present invention Figure 1 Another embodiment of the process flow diagram of S104; Figure 7 For the present invention Figure 6 A schematic diagram of an embodiment of S603; Figure 8 For the present invention Figure 1 A schematic diagram of an embodiment of S106; Figure 9 A schematic diagram of an embodiment of the cross-modal image-text matching system based on order consistency provided by the present invention; Figure 10 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] In the description of the embodiments of the present invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0021] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] This invention provides a cross-modal image-text matching method, system, and electronic device based on order consistency, which are described below.

[0024] Figure 1 This is a schematic flowchart of an embodiment of the cross-modal image-text matching method based on order consistency provided by the present invention, as shown below. Figure 1 As shown, cross-modal image-text matching methods based on ranking consistency include: S101. Obtain a training set, which contains multiple sample pairs consisting of image samples and text samples.

[0025] It should be noted that the training set can be selected from publicly available datasets, such as image-text pairs from Flickr30K or MS-COCO datasets, or from image and text data crawled from the internet. Obtaining a training set containing both image and text modalities provides the foundation for training the cross-modal alignment model. The training set contains multiple image-text pairs (i.e., sample pairs), each containing one image sample and one text sample; at least some sample pairs exhibit noisy correspondences.

[0026] S102. Based on the bidirectional cross-modal retrieval results of each sample pair between images and text, evaluate the semantic matching confidence of each sample pair.

[0027] It should be noted that for each sample pair in the training set, cross-modal retrieval is performed in both the image-to-text and text-to-image directions to obtain bidirectional cross-modal retrieval results. The bidirectional cross-modal retrieval results include one cross-modal retrieval result obtained by searching the text database using the image sample in the sample pair as the query condition in the image-to-text retrieval direction; and another cross-modal retrieval result obtained by searching the image database using the text sample in the sample pair as the query condition in the text-to-image retrieval direction. Based on this, according to the bidirectional cross-modal retrieval results corresponding to each sample pair, the semantic matching confidence level between the image sample and the text sample in each sample pair is evaluated, i.e., semantic matching confidence. Semantic matching confidence is used to characterize the degree of semantic association between the image sample and the text sample. The higher the semantic matching confidence, the more semantically consistent the image content and text description contained in the sample pair are.

[0028] S103. Select a candidate set from the training set based on the semantic matching confidence.

[0029] It should be noted that after calculating the semantic matching confidence score for each sample pair, all sample pairs need to be initially screened to distinguish between potentially matching and clearly mismatched pairs, thereby reducing the computational cost of subsequent fine-grained evaluation. Therefore, for all sample pairs in the training set, the semantic matching confidence score is used as the quantitative criterion for judging sample quality. Sample pairs that meet the semantic matching confidence score criteria are selected from the training set according to the preset screening criteria, and these selected sample pairs constitute the candidate set.

[0030] S104. Calculate the first ranking similarity of each sample pair in the candidate set in the text retrieval direction, and calculate the second ranking similarity of each sample pair in the candidate set in the image retrieval direction.

[0031] It should be noted that for each sample pair in the candidate set, a first ranking similarity is calculated from the text retrieval perspective. This means that using samples from the sample pair as retrieval criteria, a search is performed in the text database to obtain a sequence ranked by semantic similarity. A second ranking similarity is calculated from the image retrieval perspective. This means using samples from the same sample pair as retrieval criteria, a search is performed in the image database to obtain another sequence ranked by semantic similarity. Ranking similarity refers to comparing the ranking results obtained from cross-modal retrieval with the ranking results obtained from intra-modal retrieval, measuring the degree of consistency between the two.

[0032] S105. Based on the first ranking similarity and the second ranking similarity, calculate the final weight of the sample pair, wherein the final weight is positively correlated with the semantic matching degree of the sample pair.

[0033] It should be noted that the arithmetic mean of the calculated first-rank similarity and second-rank similarity is used as the final weight of the sample pair. The formula for calculating the final weight is as follows:

[0034] in, For the first i a sample pair The final training weights, The first ranking similarity (i.e., the normalized cumulative gain after deduction). The second ranking similarity (i.e., the normalized cumulative gain after deduction).

[0035] when A higher value indicates that the sample pair is... A highly consistent neighborhood structure in cross-modal retrieval implies strong semantic matching. When it is lower, it indicates that the sample pair Poor consistency in ranking across the two search directions suggests that the sample pair may be only partially relevant or completely mismatched.

[0036] S106. Train the sample pairs in the candidate set according to the final weights and update the parameters of the cross-modal alignment model.

[0037] It should be noted that after calculating the final weight of each sample pair in the candidate set, these weights are used to guide the training process of the cross-modal alignment model. All the above steps are executed iteratively until the cross-modal alignment model converges, so that the cross-modal alignment model can learn from high-quality sample pairs first when updating parameters, thereby updating the parameters of the cross-modal alignment model.

[0038] S107. Input the data to be matched into the updated model to obtain cross-modal features, and obtain the image-text matching result corresponding to the data to be matched based on the similarity between the cross-modal features.

[0039] It should be noted that once the model is trained, it can be applied to actual cross-modal image-text matching tasks. For any given data to be matched (e.g., at least one of a query image and query text), it is input into the updated cross-modal alignment model. This model, based on the cross-modal semantic alignment capabilities learned during the training phase, extracts the image cross-modal features corresponding to the query image and the text cross-modal features corresponding to the query text, respectively. Then, the similarity between the image cross-modal features and the text cross-modal features is calculated, and the image-text matching result is determined based on the similarity level. For example, it outputs a sorted list of texts most semantically relevant to the query image, or a sorted list of images most semantically relevant to the query text.

[0040] In summary, the cross-modal image-text matching method based on ranking consistency provided in this invention first calculates the bidirectional contrastive loss for each sample pair and uses a Gaussian mixture model for coarse filtering to eliminate clearly mismatched sample pairs. This avoids the cross-modal alignment model learning false cross-modal relevance from obviously erroneous sample pairs, effectively reducing training interference caused by noise correspondences. Furthermore, by calculating the first and second bidirectional ranking similarities of each sample pair in the candidate set in both the text retrieval and image retrieval directions to evaluate its global semantic matching degree, it can accurately distinguish between fully aligned sample pairs and only partially related sample pairs, avoiding interference from misleading information in partially related sample pairs, and significantly improving the retrieval accuracy and robustness of the cross-modal alignment model on noisy training data. Furthermore, by weighting the sample pairs in the candidate set based on ranking similarity and iteratively executing the above process, the model can be guided to gradually concentrate its learning resources on sample pairs with high semantic matching during training, thereby significantly reducing the retrieval performance degradation caused by noise correspondence. In practical applications of image-text matching, the model can pay more attention to the alignment between images and text at the global semantic level, thereby significantly improving the accuracy and quality of cross-modal image-text matching.

[0041] In some embodiments of the present invention, such as Figure 2 As shown, step S102 includes: S201. Extract the image feature vector of the image sample and extract the text feature vector of the text sample.

[0042] It should be noted that the CLIP (Contrastive Language-Image Pre-Training) model is used as the backbone network of the cross-modal alignment model. This model includes an image encoder and a text encoder. For each sample pair in the training set, the image sample is input into the image encoder to extract the image feature vector, and the text sample is input into the text encoder to extract the text feature vector. Both the image feature vector and the text feature vector are fixed-dimensional real-valued vectors located in the common semantic space of the cross-modal alignment model.

[0043] S202. Calculate the similarity between the image feature vector and the text feature vector.

[0044] It should be noted that: given a training set ,in, For the first One sample pair, (1 indicates a match, 0 indicates a non-match). The noisy correspondence problem refers to the mislabeling of partially or completely unrelated pairs as matches. After extracting the image feature vector and text feature vector, the similarity between them is calculated to measure the degree of matching between the image sample and the text sample in the common semantic space. This embodiment uses cosine similarity as the similarity metric, and its calculation formula is as follows:

[0045] in, For the input image sample, For the input text sample, For image samples and text samples Cosine similarity score between them For image samples Encoded image feature vector For text samples Encoded text feature vector It is the L2 norm (i.e., the magnitude) of the image feature vector. It is the L2 norm (i.e., the modulus) of the text feature vector.

[0046] S203. Calculate the first posterior probability that the image sample retrieves the corresponding text sample and the second posterior probability that the text sample retrieves the corresponding image sample based on the similarity. The first posterior probability and the second posterior probability constitute the bidirectional cross-modal retrieval result.

[0047] It should be noted that: image samples based on similarity calculation in the image-to-text retrieval direction I i Able to retrieve corresponding text samples T i The first posterior probability is calculated based on the similarity of text samples in the text-to-image retrieval direction. T i Able to retrieve corresponding image samples I i The second posterior probability is used to construct the bidirectional cross-modal retrieval result for the sample pair. The bidirectional cross-modal retrieval result characterizes the semantic association strength between the image and text corresponding to the sample pair from two directions: image retrieval of text and text retrieval of image. The higher the posterior probability in both directions, the higher the retrieval accuracy of the sample pair in both retrieval directions, that is, the higher the semantic matching degree between the image and text.

[0048] S204. Obtain the first contrast loss based on the first posterior probability, and obtain the second contrast loss based on the second posterior probability.

[0049] It should be noted that after obtaining the first posterior probability, a negative logarithmic operation is performed on it to obtain the first contrastive loss in the image-to-text direction. Specifically, in the image-to-text retrieval direction, using image samples... I i As a query condition, calculate its comparison with all text samples in the text database. T i The similarity between them was then calculated. Subsequently, the softmax normalization method in the InfoNCE (Info Noise-Contrastive Estimation) loss function was used to normalize the image samples. I i With all text samples in the text library T i The similarity is converted into a probability distribution, where the image samples I i Corresponding text sample T i The proportion of similarity between two images in the total similarity is the first posterior probability. A higher first posterior probability indicates a stronger similarity among image samples under the current model. I i Able to accurately retrieve corresponding text samples from a text database T i The greater the likelihood of a match, the better the alignment of the sample pair in the image-to-text retrieval direction. Specifically, a lower first contrast loss value indicates a better fit of the model to the sample pair in the image-to-text retrieval direction; conversely, a higher first contrast loss value indicates that the model has difficulty aligning the image sample pair in that direction. I i Corresponding text samples T i Perform a match.

[0050] Similarly, after obtaining the second posterior probability, a negative logarithmic operation is performed on it to obtain the second contrast loss in the text-to-image direction. Specifically, in the text-to-image retrieval direction, using text samples... T i As a query condition, calculate its comparison with all image samples in the image database. I i The similarity between them was then assessed. Subsequently, the same softmax normalization method was used to normalize the text samples. T i Compared with all image samples in the image library I i The similarity is converted into a probability distribution, where the text samples T i With corresponding image samples I iThe proportion of similarity between two text samples in the total similarity is the second posterior probability. A higher second posterior probability indicates a stronger similarity among text samples under the current model. T i Able to accurately retrieve corresponding image samples from image databases I i The greater the likelihood of a match, the better the alignment of the sample pair in the text-to-image retrieval direction. Specifically, a lower second contrast loss value indicates a better fit of the model to the sample pair in the text-to-image retrieval direction; conversely, a higher second contrast loss value indicates that the model has difficulty matching the text sample with the corresponding image sample in that direction.

[0051] In summary, by utilizing the memory effect of deep neural networks (DNNs typically learn the correct labels first and then fit the noisy labels), for each pair Calculate the loss value for two-way comparison:

[0052] in, The first contrast loss in the image-to-text direction, The second contrast loss is for the text-to-image direction. For image samples Text samples were retrieved correctly. The first posterior probability, For text-based samples Image samples were retrieved correctly. The second posterior probability.

[0053] S205. Evaluate the semantic matching confidence of each sample pair based on the first contrast loss and the second contrast loss.

[0054] It should be noted that after obtaining the first and second contrastive losses for each sample pair, the semantic matching confidence of each sample pair is evaluated based on the contrastive losses in these two directions. The first and second contrastive losses together constitute the quantitative basis for evaluating the semantic matching confidence of the sample pair. This is achieved by comprehensively considering the contrastive losses in both directions.

[0055] In this embodiment, by calculating the first contrast loss in the image-to-text direction and the second contrast loss in the text-to-image direction, and evaluating the semantic matching confidence of each sample pair, the cross-modal alignment quality of each sample pair can be comprehensively evaluated from a bidirectional retrieval perspective. This avoids bias caused by evaluation in a single direction and improves the accuracy of judging the matching confidence of sample pairs. Furthermore, by calculating the posterior probability and using it to calculate the corresponding contrast loss, well-matched sample pairs have lower loss values, and poorly matched sample pairs have higher loss values. This improves the accuracy of sample selection based on the loss distribution, reduces the dependence of noise correspondence recognition on manually labeled data, and enhances the automation and adaptability of the training process in large-scale multimodal data scenarios.

[0056] In some embodiments of the present invention, such as Figure 3 As shown, step S103 includes: S301. The first distribution of the first contrast loss and the second distribution of the second contrast loss for each sample pair are fitted using the Gaussian mixture model, wherein the Gaussian mixture model includes a low loss component and a high loss component.

[0057] It should be noted that, to further quantify the semantic matching confidence into specific probability values ​​for easier filtering, this embodiment uses a one-dimensional two-component Gaussian Mixture Model (GMM) to model the distributions of the first and second contrastive losses for each sample pair. A GMM is a probabilistic model used to represent a population containing multiple sub-distributions, capable of automatically classifying samples into different groups based on the magnitude of their loss values ​​without prior knowledge of their class affiliation. The probability density function of the GMM is shown below:

[0058] in, Loss value The probability density distribution in a GMM The loss value of the input GMM (i.e., the value calculated in formula (2)) is used as the input. or ), For the index of the Gaussian component, For the first Mixed weights of Gaussian components (prior probabilities). Let be a Gaussian probability density function. For the first The mean of each Gaussian component (a component with a larger mean corresponds to a higher loss for mismatched samples, and a component with a smaller mean corresponds to a lower loss for matched samples). For the first The variance of each Gaussian component (used to measure the dispersion of sample loss within that component).

[0059] When fitting the loss distribution using a two-component Gaussian mixture model, due to the characteristic of deep neural networks learning simple patterns before complex patterns during training, well-matched sample pairs typically have lower contrastive losses, while explicitly mismatched sample pairs typically have higher contrastive losses. Therefore, of the two Gaussian components obtained after fitting, the component with the lower mean corresponds to matched or partially matched sample pairs and is called the low-loss component; the component with the higher mean corresponds to explicitly mismatched sample pairs and is called the high-loss component. In this embodiment, the Expectation-Maximization (EM) algorithm is used to independently fit the loss distributions in the image-to-text direction and the text-to-image direction to determine the parameters of each Gaussian component (including the mixture weights). mean and variance ).

[0060] S302. For each sample pair, calculate the first probability that the first contrast loss of the sample pair belongs to the low loss component and the second probability that the second contrast loss belongs to the low loss component.

[0061] It should be noted that: for the image-to-text direction, the first contrast loss of the sample pair is calculated based on the GMM distribution of the fitted first contrast loss. The first posterior probability belongs to the low-loss component (the Gaussian component with a low mean). For the text-to-image direction, the second contrastive loss for that sample pair is calculated based on the fitted GMM distribution of the second contrastive loss. The second posterior probability belongs to the low loss component. The higher the posterior probability, the closer the contrastive loss of the sample pair in that direction is to the typical loss feature of the matched sample, that is, the better the cross-modal alignment quality in that direction; conversely, the lower the posterior probability, the closer the contrastive loss of the sample pair in that direction is to the typical loss feature of the mismatched sample, that is, the worse the cross-modal alignment quality in that direction.

[0062] S303. Select sample pairs whose first probability is greater than a preset threshold and whose second probability is greater than a preset threshold into the candidate set.

[0063] It should be noted that a preset threshold (e.g., 0.5) is set to determine whether the sample pair has good cross-modal alignment quality in both directions. For each sample pair... The algorithm determines whether the first posterior probability of a sample pair in the image-to-text direction is greater than a preset threshold, and simultaneously determines whether the second posterior probability of a sample pair in the text-to-image direction is greater than a preset threshold. A sample pair is considered valid if it satisfies both conditions simultaneously in both directions. This indicates that the sample pair exhibits low contrastive loss in both retrieval directions and high cross-modal semantic consistency, thus it is selected for inclusion in the candidate set. The expression for the candidate set is shown below:

[0064] in, For the first The loss for each sample in the image-to-text direction is the posterior probability that it belongs to the low-mean component (matching component). For the first The loss for each sample in the text-to-image direction is the posterior probability that it belongs to the low-mean component (matching component). The preset threshold hyperparameter is used. The candidate set is the set of sample indices retained after coarse filtering by GMM, i.e., only when a sample exhibits low loss in both directions ( and Only then will it enter the subsequent reweighted candidate set. The remaining samples are considered as clearly mismatched pairs and subsequently assigned zero weight. This completes the coarse filtering based on the loss distribution (GMM filtering).

[0065] In this embodiment, by fitting the contrast loss distribution of each sample pair in both directions using a Gaussian mixture model, matching and non-matching samples can be adaptively distinguished based on the memory effect of deep neural networks during training. This eliminates the need for manual annotation or external supervision signals, reducing reliance on clean labeled data and improving the automation and adaptability of noise correspondence identification. Furthermore, by calculating the posterior probability of each sample pair belonging to the low-loss component in both directions and setting a preset threshold for bidirectional screening, only sample pairs exhibiting low-loss characteristics in both retrieval directions are selected into the candidate set. This avoids bias caused by unidirectional evaluation, improves the overall matching confidence of sample pairs in the candidate set, and effectively reduces the risk of mistakenly selecting partially relevant or non-matching sample pairs into the candidate set. Furthermore, by excluding sample pairs outside the candidate set from subsequent fine-grained evaluation and weighted training and assigning them zero weight, the model avoids learning false cross-modal correlations from explicitly non-matching sample pairs, reducing training interference caused by noise correspondences.

[0066] In some embodiments of the present invention, such as Figure 4 As shown, step S104 includes: S401. Based on the image samples, a search is performed in the text library to obtain a first actual sorted list.

[0067] It should be noted that for each sample pair in the candidate set, cross-modal retrieval is performed in the text database using the image sample as the query condition. Specifically, the cross-modal similarity between the image feature vector of the image sample and the text feature vectors of all text samples in the text database is calculated to obtain a similarity score between the image sample and each text sample. All text samples are then sorted in descending order of similarity score to obtain the first actual ranking list. This first actual ranking list reflects which text samples the image sample can retrieve under the current cross-modal alignment model and the ranking of these text samples and their semantic relevance.

[0068] S402. Based on the text sample, a search is performed in the text database to obtain a first ideal sorted list.

[0069] It should be noted that for each sample pair in the candidate set, an intra-modal search is performed using the text sample as the query within the same text database. Specifically, the intra-modal similarity between the text feature vector of the text sample and the text feature vectors of all text samples in the text database is calculated to obtain a similarity score between the text sample and each other. All text samples are then sorted in descending order of similarity score to obtain a first ideal ranking list. The first ideal ranking list represents the ranking of the text samples most similar to the original text sample within the semantic space of the text modality itself.

[0070] S403. Calculate the first sorting similarity based on the first actual sorting list and the first ideal sorting list.

[0071] It should be noted that Normalized Discounted Cumulative Gain (NDCG) is used as a measure of ranking similarity. The first ranking similarity between the first actual ranking list and the first ideal ranking list is calculated.

[0072] In this embodiment, by performing cross-modal retrieval in the text library using image samples and intramodal retrieval in the same text library using corresponding text samples, a first actual ranking list and a first ideal ranking list are obtained. The normalized first ranking similarity is calculated by using the first actual depreciation cumulative gain and the first ideal depreciation cumulative gain. This can eliminate the evaluation bias caused by the difference in ideal ranking quality between different query samples, making the ranking similarity comparable between different sample pairs. This improves the consistency evaluation ability of ranking similarity among samples and enhances the uniformity and accuracy of training weight allocation.

[0073] In some embodiments of the present invention, such as Figure 5 As shown, step S403 includes: S501. Obtain the first intramodal similarity between the top K texts in the first actual sorting list and the text samples. Calculate the corresponding first gain value based on the first intramodal similarity of each ranking position. Calculate the first actual loss cumulative gain based on the first gain value of each ranking position and the position loss factor. S502. Obtain the second intramodal similarity between the top K texts in the first ideal ranking list and the text samples. Calculate the corresponding second gain value based on the second intramodal similarity corresponding to each ranking position. Calculate the first ideal loss cumulative gain based on the second gain value of each ranking position and the position loss factor. S503. Calculate the first sorting similarity based on the first actual cumulative loss gain and the first ideal cumulative loss gain.

[0074] It should be noted that K is a positive integer. If the image... I i and text T i Describing the same semantics, they should occupy similar positions in the feature space. Therefore, using I i The sorting generated by the query to retrieve other texts, and the use of T i The ranking generated by retrieving other texts should be highly consistent. Based on this idea, this stage evaluates the ranking consistency of each candidate sample pair from two dimensions: text retrieval direction and image retrieval direction. The text retrieval direction evaluates the ranking quality when retrieving text with an image as the query, while the image retrieval direction evaluates the ranking quality when retrieving images with text as the query. These two directions complement each other, forming a bidirectional ranking consistency evaluation framework. The ranking consistency evaluation in the text retrieval direction requires comparing two types of ranking lists: one is the cross-modal actual ranking list obtained by querying the text library based on image samples, and the other is the intra-modal ideal ranking list obtained by querying the same text library based on text samples. Specifically, the first actual ranking (cross-modal) is based on image... I i The similarity ranking of all texts generated from a text library across a modality (image to text) is denoted as . The first ideal ranking (intramodal) is based on text. T i The similarity ranking of all texts generated from a text library within a single modality (text-to-text) is denoted as . .

[0075] The formula for calculating (cumulative gain due to loss) is as follows:

[0076] in, To use images in the text retrieval direction (image → text) As the first actual cumulative gain at the time of query; The truncation depth hyperparameter represents the top-ranked results in the evaluation search. One location; This is the index for the ranking position in the search results list. The first in the actual sorting The result represents the image. Search the text library for the returned text. A text, To calculate anchor text and the retrieved first First-modal similarity (text-to-text similarity) between texts. This is the first gain value. This is the position loss factor.

[0077] Thus, using text-to-text similarity As the relevance score of the j-th text in the first actual ranking, i.e., the evaluation image Did you find any results related to...? Texts with similar semantics. Among them, The formula for calculating (ideal DCG) is as follows:

[0078] in, To use images in the text retrieval direction (image → text) As the first ideal discount cumulative gain when querying; The truncation depth hyperparameter represents the top-ranked results in the evaluation search. One location; This is the index for the ranking position in the search results list. The first in the ideal order The result represents the text. Search the text library for the returned text. A text, To calculate anchor text And the ideal retrieved first Intramodal similarity (text-to-text similarity) between texts. This is the second gain value. This is the location loss factor.

[0079] The formula for calculating (normalized loss cumulative gain) is as follows:

[0080] in, The first ranking similarity (i.e., the normalized cumulative gain after loss) is used to measure the degree of similarity or consistency between the first actual ranking and the first ideal ranking. The value range of the first ranking similarity is [0, 1]. When the value is 1, it means that the first actual ranking list is completely consistent with the first ideal ranking list, that is, the cross-modal retrieval ranking has achieved the same ideal ranking effect as the intra-modal retrieval. When the value approaches 0, it means that the first actual ranking list is seriously inconsistent with the first ideal ranking list, that is, the cross-modal retrieval ranking deviates from the ideal semantic ranking. Therefore, the higher the ranking similarity, the closer the ranking result obtained by cross-modal retrieval with image samples as queries is to the ideal ranking obtained by intra-modal retrieval with corresponding text samples as queries, that is, the higher the semantic matching quality of the sample pair in the text retrieval direction.

[0081] In this embodiment, the above calculation process improves the accuracy of ranking similarity in evaluating the semantic matching degree of samples, enhances the comparability of ranking similarity between different sample pairs, and thus improves the rationality and consistency of training weight allocation.

[0082] In some embodiments of the present invention, such as Figure 6 As shown, step S403 further includes: S601. Based on the text sample, a search is performed in the image library to obtain a second actual sorted list.

[0083] It should be noted that for each sample pair in the candidate set, a cross-modal retrieval is performed in the image database using the text sample as the query condition. Specifically, the cross-modal similarity between the text feature vector of the text sample and the image feature vectors of all image samples in the image database is calculated to obtain the similarity score between the text sample and each image sample. All image samples are then sorted in descending order of similarity score to obtain the second actual ranking list. The second actual ranking list Rt2i(Ti) reflects which image samples the text sample can retrieve under the current cross-modal alignment model and the ranking of these image samples and their semantic relevance.

[0084] S602. Based on the image samples, a second ideal sorting list is obtained by searching the image library.

[0085] It should be noted that for each sample pair in the candidate set, an intra-modal search is performed within the same image database using the image sample as the query. Specifically, the intra-modal similarity between the image feature vector of the image sample and the image feature vectors of all image samples in the image database is calculated to obtain a similarity score between the image sample and each other. All image samples are then sorted in descending order of similarity score to obtain a second ideal ranking list. The second ideal ranking list represents the ranking of the image samples most similar to the image sample within the semantic space of the image modality itself, reflecting the positional relationship of the image sample in the global image semantic manifold.

[0086] S603. Calculate the second sorting similarity based on the second actual sorting list and the second ideal sorting list.

[0087] It should be noted that Normalized Discounted Cumulative Gain (NDCG) is used as a measure of ranking similarity. The second ranking similarity between the second actual ranking list and the second ideal ranking list is calculated.

[0088] In this embodiment, by performing cross-modal retrieval in the image library using text samples and intramodal retrieval in the same image library using corresponding image samples, a second actual ranking list and a second ideal ranking list are obtained. The normalized second ranking similarity is calculated using the second actual depreciation cumulative gain and the second ideal depreciation cumulative gain, making the ranking similarity comparable between different sample pairs. This improves the consistency evaluation ability of ranking similarity among samples and enhances the uniformity and accuracy of training weight allocation.

[0089] In some embodiments of the present invention, such as Figure 7 As shown, step S603 includes: S701. Obtain the intra-modal similarity between the top K images in the second actual ranking list and the image samples. Calculate the corresponding third gain value based on the intra-modal similarity of each ranking position. Calculate the second actual loss cumulative gain based on the third gain value of each ranking position and the position loss factor. S702. Obtain the intra-modal similarity between the top K images in the second ideal ranking list and the image samples. Calculate the corresponding fourth gain value based on the intra-modal similarity of each ranking position. Calculate the second ideal loss cumulative gain based on the fourth gain value of each ranking position and the position loss factor. S703. Calculate the second sorting similarity based on the second actual cumulative loss gain and the second ideal cumulative loss gain.

[0090] It should be noted that, similarly, in image retrieval, two types of ranking lists need to be compared: one is the cross-modal actual ranking list obtained by querying the image database based on text samples, and the other is the intra-modal ideal ranking list obtained by querying the same image database based on image samples. Specifically, the second actual ranking (cross-modal) is based on text... T i The similarity ranking of all images in the image library across a modal (text-to-image) is denoted as . The second ideal ranking (intramodal) is based on the image. I i The similarity ranking within a modality (image-to-image) of all images in the image library is denoted as . .

[0091] The formula for calculating (cumulative gain due to loss) is as follows:

[0092] in, To use text in the image retrieval direction (text → image) As the second actual cumulative gain of loss during the query; The truncation depth hyperparameter represents the top-ranked results in the evaluation search. One location; This is the index for the ranking position in the search results list. For the second actual sorting The result represents the text. Search the image library for the returned image. One image, To calculate anchor point image and the retrieved first Third-modal similarity (image-image similarity) between images. This is the third gain value. This is the location loss factor.

[0093] Thus, using image-image similarity As the second actual ordering The relevance score of a bit image, i.e., the assessment Did you find any results related to...? Images with similar semantics. Among them, The formula for calculating (ideal DCG) is as follows:

[0094] in, To use text in the image retrieval direction (text → image) As the second ideal loss accumulation gain during query; The truncation depth hyperparameter represents the top-ranked results in the evaluation search. One location; This is the index for the ranking position in the search results list. The second ideal ordering The result represents the image. Search the image library for the returned image. One image, To calculate anchor point image And the ideal retrieved first One image The similarity between images within the fourth modality (image-image similarity). This is the fourth gain value. This is the location loss factor.

[0095] The formula for calculating (normalized loss cumulative gain) is as follows:

[0096] in, The second ranking similarity (i.e., the normalized cumulative gain after loss) measures the degree of similarity or consistency between the second actual ranking and the second ideal ranking. The value range of the second ranking similarity is [0, 1]. When the value is 1, it indicates that the second actual ranking list is completely consistent with the second ideal ranking list, meaning that the cross-modal retrieval ranking has achieved the same ideal ranking effect as intra-modal retrieval. When the value approaches 0, it indicates that the second actual ranking list is seriously inconsistent with the second ideal ranking list, meaning that the cross-modal retrieval ranking deviates from the ideal semantic ranking. Therefore, the higher the ranking similarity, the closer the ranking result obtained by cross-modal retrieval with text samples as queries is to the ideal ranking obtained by intra-modal retrieval with corresponding image samples as queries, meaning that the sample pair has a higher semantic matching quality in the image retrieval direction.

[0097] In this embodiment, the above calculation process improves the accuracy of ranking similarity in evaluating the semantic matching degree of samples, enhances the comparability of ranking similarity between different sample pairs, and thus improves the rationality and consistency of training weight allocation.

[0098] In some embodiments of the present invention, such as Figure 8 As shown, step S106 includes: S801. Determine the contribution of each sample pair in the candidate set during training based on the final weight.

[0099] It should be noted that for each sample pair in the candidate set, its final weight is used as its contribution during training. A higher weight indicates a greater influence on model parameter updates. Sample pairs outside the candidate set have zero contribution and no influence on model parameter updates. To integrate contribution with the loss function, for each sample pair in the candidate set, a soft label is generated based on its final weight, and the soft labels for sample pairs outside the candidate set are set to zero. Based on these rules, the... i The soft labels for each sample pair can be uniformly represented as follows:

[0100] in, For the first i a sample pair Soft labels are used to replace the original hard labels in the loss function. For indicator functions, For candidate set, For the first i a sample pair The final training weights.

[0101] S802. Weight the corresponding contrast loss according to the contribution, and update the parameters of the cross-modal alignment model according to the weighted contrast loss.

[0102] It should be noted that after determining the contribution of each sample pair in the candidate set, the corresponding contrastive loss is weighted using the contribution of each sample pair, and the parameters of the cross-modal alignment model are updated based on the weighted contrastive losses. Specifically, for each sample pair in the training set, its log probability loss in the image-to-text direction is... The log probability loss in the text-to-image direction is The logarithmic probability loss for each sample pair in the two directions mentioned above is summed, and then multiplied by the soft label of that sample pair. The weighted loss contribution of each sample pair is obtained. Then, the weighted loss contributions of all sample pairs are summed and taken as a negative average to obtain the total loss value L. The formula for calculating the total loss value is as follows:

[0103] in, Here, represents the final total loss value for model training, and represents the total number of sample pairs in the training set. For image samples Able to correctly retrieve text samples The first posterior probability, For text samples Able to correctly retrieve image samples The second posterior probability, For image samples To text sample The first logarithmic probability loss in the direction, For text samples To image samples The second logarithmic probability loss in the direction.

[0104] After obtaining the total loss value, gradient descent is used to backpropagate the total loss value, calculating the gradient of the total loss value with respect to the parameters of the cross-modal alignment model. The parameters of the image encoder and text encoder are then updated along the gradient descent direction, enabling the model to better align the image and text semantics in high-quality sample pairs in the next iteration. After completing this round of parameter updates, steps S101 to S106 are repeated until the cross-modal alignment model converges. In each iteration, the candidate set and final weights are dynamically updated based on the current model state, allowing the model to gradually adapt to and correct noise correspondences during training.

[0105] In this embodiment, the learning priority of the model for high-quality sample pairs is increased, which enhances the convergence speed and retrieval accuracy of the cross-modal alignment model, while reducing the risk of retrieval performance degradation caused by noise correspondence.

[0106] This invention utilizes the memory effect of deep neural networks to calculate the bidirectional contrastive loss for each sample pair. A two-component Gaussian mixture model (GMM) is used to fit the loss distribution, and the expectation-maximization (EM) algorithm is used to distinguish between matching and non-matching samples. A candidate sample set C with low losses in both directions is selected, and the remaining samples are marked as non-matching and assigned zero weight. For sample pairs in the candidate set C, the ranking consistency in the text retrieval direction and the image retrieval direction are calculated separately. By comparing the NDCG scores of cross-modal retrieval ranking and intra-modal retrieval ranking, the semantic consistency between image and text samples in the global data manifold is evaluated. The average of the NDCG scores in the two directions is taken as the final weight of the sample. The calculated weights are applied to the InfoNCE contrastive loss function, and the samples in the candidate set are reweighted and trained according to their weights to achieve noise-robust cross-modal alignment learning.

[0107] This invention overcomes the shortcomings of existing methods that rely on model similarity prediction through a two-stage strategy: coarse filtering using GMM + fine reweighting using NDCG (see step S104 and its corresponding embodiment). Even if a single similarity score is unreliable, the metric based on ranking consistency remains stable because ranking consistency examines the relative positional relationship of samples within the global neighborhood structure, thus eliminating the reliance on model similarity prediction. Even if two sample pairs have similarity scores, RCR can distinguish them by comparing their ranking consistency with the global data structure. Well-aligned pairs will produce highly consistent cross-modal and intra-modal retrieval rankings; while partially aligned pairs, even if they obtain high similarity scores, will have low ranking consistency, thus accurately distinguishing between well-aligned and partially aligned sample pairs. In addition, this invention, based on the NDCG metric, naturally considers the relationship between samples and multiple other data points, capturing the global manifold structure of the data. This global perspective makes the reweighting decision more robust and less susceptible to local noise interference. This invention rapidly eliminates obviously mismatched samples through a coarse filtering stage using GMM, reducing subsequent computational overhead. A fine reweighting stage using NDCG then meticulously evaluates the remaining candidates, ensuring the accuracy of the reweighting. This achieves state-of-the-art performance in both synthetic noise (0%-60% noise rate) and real-world noise environments, with training time only 2.2 times that of baseline methods (4.7 times for NPC, 7.4 times for GLP), demonstrating high computational efficiency. Even on clean datasets without artificially added noise, performance is improved (indicating the existence of partial matches or fuzzy pairs in real-world data). This invention can be seamlessly integrated into existing cross-modal retrieval frameworks without modifying the model architecture.

[0108] To verify the effectiveness of this approach, experiments were conducted on several benchmark datasets, including synthetic noise datasets (Flickr30K, MS-COCO, simulating noise rates of 0%-60% by randomly shuffling titles) and real-world noise datasets (CC120K, natural noise crawled from the internet). On these datasets, the proposed approach (RCR) was compared and evaluated with several existing state-of-the-art methods. First, the robustness of the proposed approach under different noise rates was verified on the synthetic noise dataset. Then, the effectiveness of the proposed approach in practical applications was verified on the real-world noise dataset. Specific comparison results are as follows: (1) Comparison with advanced methods (synthetic noise, Flickr 30K) is shown in Table 1 below.

[0109]

[0110] Table 1 shows the performance comparison on the Flickr30K dataset under different noise rates. As shown in Table 1, RCR significantly outperforms the baseline method across all noise rates. Notably, RCR also shows a significant improvement at 0% noise (i.e., the original clean data), indicating that even "clean" datasets contain partially matched or ambiguous pairs, and the global ranking consistency metric naturally reduces the weight of these pairs.

[0111] (2) The comparison with advanced methods (synthetic noise, MS-COCO 1K) is shown in Table 2 below.

[0112]

[0113] Table 2 shows the performance comparison on the MS-COCO 1K dataset at different noise rates. As shown in Table 2, RCR also maintains the best performance across all noise rates on the MS-COCO dataset, verifying the method's ability to generalize across datasets.

[0114] (3) The comparison of real noise datasets (CC120K) is shown in Table 3 below.

[0115]

[0116] Table 3 shows the performance comparison on the real-world noise dataset CC120K. As shown in Table 3, on the real-world noise dataset CC120K, RCR achieves the best performance across all metrics, with rSum reaching 517.3, significantly outperforming GLP (515.0) and NPC (513.6).

[0117] This invention can be applied to deep learning scenarios requiring large-scale multimodal data alignment, such as image-text retrieval, video-text retrieval, multimodal content understanding, visual question answering, and cross-modal recommendation systems. It is particularly suitable for training models with noisy correspondences in multimodal data crawled from the internet. For example, an internet news platform generates approximately 500,000 news articles daily, each containing a news image and a text description (title or summary). The platform aims to use this image-text data to train a cross-modal retrieval model, enabling "searching for news by image" and "searching for images by text." The platform collects image-text data from public news websites using a web crawler system, with each data sample consisting of an image-text pair. However, due to the following reasons, the collected data contains a large number of noisy correspondences, including sources such as alt text unrelated to the image, titles only partially related to the image, and semantic inconsistencies between the image and text. In such cases, if all the collected image-text data is directly used to train the cross-modal alignment model, the model will force mismatched or only partially matched images and texts to be brought closer to the same semantic space, thus learning a large number of spurious cross-modal relevances. For example, the model might incorrectly identify "the president of a certain country" as "the president of a certain country". Figure 1There is a semantic relationship between them, which seriously reduces the performance of subsequent retrieval tasks.

[0118] This invention provides a cross-modal sample reweighting method (RCR) based on ranking consistency. It estimates the semantic matching degree of samples by comparing the NDCG ranking consistency of cross-modal and intra-modal retrieval results, eliminating reliance on the model's own similarity prediction. In the first stage, this invention uses a two-component Gaussian mixture model (GMM) to quickly filter obviously mismatched sample pairs based on the loss distribution. In the second stage, it performs fine-grained reweighting of the remaining candidate samples using NDCG ranking consistency, balancing efficiency and accuracy. This invention uses a semantic matching degree measurement method based on the global data manifold structure: it uses the relative ranking position of samples in the global neighborhood structure rather than local similarity scores to evaluate matching quality, maintaining stable reweighting decisions even if individual similarity predictions are unreliable. This invention achieves fine-grained differentiation between well-aligned and partially aligned sample pairs: even with similarity scores, RCR can identify partially aligned pairs through ranking consistency differences and reduce their weights, achieving more accurate soft label estimation.

[0119] To better implement the cross-modal image-text matching method based on order consistency in the embodiments of the present invention, based on the cross-modal image-text matching method based on order consistency, the corresponding method is as follows: Figure 9 As shown, this embodiment of the invention also provides a cross-modal image-text matching system 900 based on ranking consistency. The cross-modal image-text matching system 900 based on ranking consistency includes: The data acquisition module 901 is used to acquire a training set, which contains multiple sample pairs consisting of image samples and text samples; The retrieval evaluation module 902 is used to evaluate the semantic matching confidence of each sample pair based on the bidirectional cross-modal retrieval results between the images and text of each sample pair. Data filtering module 903 is used to filter candidate sets from the training set based on the semantic matching confidence. The similarity calculation module 904 is used to calculate the first ranking similarity of each sample pair in the candidate set in the text retrieval direction, and to calculate the second ranking similarity of each sample pair in the candidate set in the image retrieval direction; The weight calculation module 905 is used to calculate the final weight of the sample pair based on the first ranking similarity and the second ranking similarity, wherein the final weight is positively correlated with the semantic matching degree of the sample pair; The model update module 906 is used to train the sample pairs in the candidate set according to the final weights and update the parameters of the cross-modal alignment model. The image-text matching module 907 is used to input the data to be matched into the updated model to obtain cross-modal features, and obtain the image-text matching result corresponding to the data to be matched based on the similarity between the cross-modal features.

[0120] The cross-modal image-text matching system 900 based on order consistency provided in the above embodiments can realize the technical solutions described in the embodiments of the cross-modal image-text matching method based on order consistency. The specific implementation principles of each module or unit can be found in the corresponding content in the embodiments of the cross-modal image-text matching method based on order consistency, and will not be repeated here.

[0121] like Figure 10 As shown, the present invention also provides an electronic device 1000. The electronic device 1000 includes a processor 1001, a memory 1002, and a display 1003. Figure 10 Only some components of the electronic device 1000 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0122] In some embodiments, processor 1001 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 1002 or process data, such as the cross-modal image-text matching method based on order consistency in this invention.

[0123] In some embodiments, processor 1001 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 1001 may be local or remote. In some embodiments, processor 1001 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, internal cloud, multi-cloud, etc., or any combination thereof.

[0124] In some embodiments, memory 1002 may be an internal storage unit of electronic device 1000, such as a hard disk or memory of electronic device 1000. In other embodiments, memory 1002 may also be an external storage device of electronic device 1000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 1000.

[0125] Furthermore, the memory 1002 may include both internal storage units of the electronic device 1000 and external storage devices. The memory 1002 is used to store application software and various types of data installed on the electronic device 1000.

[0126] In some embodiments, display 1003 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 1003 is used to display information from electronic device 1000 and to display a visual user interface. Components 1001-1003 of electronic device 1000 communicate with each other via a system bus.

[0127] In one embodiment, when processor 1001 executes a cross-modal image-text matching program based on order consistency in memory 1002, the following steps may be implemented: Obtain a training set, which contains multiple sample pairs consisting of image samples and text samples; Based on the bidirectional cross-modal retrieval results of each sample pair between images and text, the semantic matching confidence of each sample pair is evaluated. A candidate set is selected from the training set based on the semantic matching confidence. Calculate a first ranking similarity for each sample pair in the candidate set in the text retrieval direction, and calculate a second ranking similarity for each sample pair in the candidate set in the image retrieval direction; Based on the first ranking similarity and the second ranking similarity, the final weight of the sample pair is calculated, and the final weight is positively correlated with the semantic matching degree of the sample pair; The parameters of the cross-modal alignment model are updated by training the sample pairs in the candidate set based on the final weights.

[0128] It should be understood that when the processor 1001 executes the cross-modal image-text matching program based on order consistency in the memory 1002, in addition to the functions mentioned above, it can also perform other functions, as detailed in the description of the corresponding method embodiments above.

[0129] Furthermore, the embodiments of the present invention do not specifically limit the type of the electronic device 1000 mentioned. The electronic device 1000 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 1000 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0130] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0131] The foregoing has provided a detailed description of the cross-modal image-text matching method, system, and electronic device based on order consistency provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A cross-modal image-text matching method based on order consistency, characterized in that, include: Obtain a training set, which contains multiple sample pairs consisting of image samples and text samples; Based on the bidirectional cross-modal retrieval results of each sample pair between images and text, the semantic matching confidence of each sample pair is evaluated. A candidate set is selected from the training set based on the semantic matching confidence. Calculate a first ranking similarity for each sample pair in the candidate set in the text retrieval direction, and calculate a second ranking similarity for each sample pair in the candidate set in the image retrieval direction; Based on the first ranking similarity and the second ranking similarity, the final weight of the sample pair is calculated, and the final weight is positively correlated with the semantic matching degree of the sample pair; The parameters of the cross-modal alignment model are updated by training the sample pairs in the candidate set based on the final weights. The data to be matched is input into the updated model to obtain cross-modal features, and the image-text matching result corresponding to the data to be matched is obtained based on the similarity between the cross-modal features.

2. The cross-modal image-text matching method based on order consistency according to claim 1, characterized in that, The evaluation of the semantic matching confidence of each sample pair based on the bidirectional cross-modal retrieval results between images and text includes: Extract the image feature vector of the image sample and extract the text feature vector of the text sample; Calculate the similarity between the image feature vector and the text feature vector; The first posterior probability of the image sample retrieving the corresponding text sample and the second posterior probability of the text sample retrieving the corresponding image sample are calculated based on the similarity. The first posterior probability and the second posterior probability constitute the bidirectional cross-modal retrieval result. The first contrast loss is obtained based on the first posterior probability, and the second contrast loss is obtained based on the second posterior probability. The semantic matching confidence of each sample pair is evaluated based on the first contrast loss and the second contrast loss.

3. The cross-modal image-text matching method based on order consistency according to claim 2, characterized in that, The step of selecting a candidate set from the training set based on the semantic matching confidence includes: The Gaussian mixture model is used to fit the first distribution of the first contrastive loss and the second distribution of the second contrastive loss for each sample pair, respectively, wherein the Gaussian mixture model includes a low loss component and a high loss component; For each sample pair, calculate the first probability that the first contrastive loss of the sample pair belongs to the low loss component and the second probability that the second contrastive loss belongs to the low loss component. Sample pairs with both the first probability and the second probability greater than the preset threshold are selected and added to the candidate set.

4. The cross-modal image-text matching method based on order consistency according to claim 1, characterized in that, The calculation of the first ranking similarity of each sample pair in the candidate set in the text retrieval direction includes: Based on the image samples, a search is performed in the text library to obtain a first actual sorted list; Based on the text sample, a first ideal sorted list is obtained by searching the text database. The first sorting similarity is calculated based on the first actual sorting list and the first ideal sorting list.

5. The cross-modal image-text matching method based on order consistency according to claim 4, characterized in that, The step of calculating the first sorting similarity based on the first actual sorting list and the first ideal sorting list includes: Obtain the first intramodal similarity between the top K texts in the first actual sorting list and the text sample; calculate the corresponding first gain value based on the first intramodal similarity corresponding to each ranking position; and calculate the first actual loss cumulative gain based on the first gain value of each ranking position and the position loss factor. Obtain the second intramodal similarity between the top K texts in the first ideal ranking list and the text sample; calculate the corresponding second gain value based on the second intramodal similarity corresponding to each ranking position; and calculate the first ideal loss cumulative gain based on the second gain value of each ranking position and the position loss factor. The first ranking similarity is calculated based on the first actual cumulative loss gain and the first ideal cumulative loss gain.

6. The cross-modal image-text matching method based on order consistency according to claim 1, characterized in that, The calculation of the second ranking similarity of each sample pair in the candidate set in the image retrieval direction includes: Based on the text sample, a second actual sorted list is obtained by searching the image library. Based on the image samples, a second ideal sorted list is obtained by searching the image library. The second sorting similarity is calculated based on the second actual sorting list and the second ideal sorting list.

7. The cross-modal image-text matching method based on order consistency according to claim 6, characterized in that, The step of calculating the second sorting similarity based on the second actual sorting list and the second ideal sorting list includes: Obtain the intra-modal similarity between the top K images in the second actual ranking list and the image samples. Calculate the corresponding third gain value based on the intra-modal similarity of each ranking position. Calculate the second actual loss cumulative gain based on the third gain value of each ranking position and the position loss factor. Obtain the intra-modal similarity between the top K images in the second ideal ranking list and the image samples. Calculate the corresponding fourth gain value based on the intra-modal similarity of each ranking position. Calculate the second ideal loss cumulative gain based on the fourth gain value of each ranking position and the position loss factor. The second ranking similarity is calculated based on the second actual cumulative loss gain and the second ideal cumulative loss gain.

8. The cross-modal image-text matching method based on order consistency according to any one of claims 1 to 7, characterized in that, The step of training the sample pairs in the candidate set according to the final weights and updating the parameters of the cross-modal alignment model includes: The contribution of each sample pair in the candidate set during training is determined based on the final weights. The corresponding contrast loss is weighted according to the contribution, and the parameters of the cross-modal alignment model are updated according to the weighted contrast loss.

9. A cross-modal image-text matching system based on ranking consistency, characterized in that, include: The data acquisition module is used to acquire a training set, which contains multiple sample pairs consisting of image samples and text samples; The retrieval evaluation module is used to evaluate the semantic matching confidence of each sample pair based on the bidirectional cross-modal retrieval results between the images and text. The data filtering module is used to filter candidate sets from the training set based on the semantic matching confidence. A similarity calculation module is used to calculate a first ranking similarity of each sample pair in the candidate set in the text retrieval direction, and to calculate a second ranking similarity of each sample pair in the candidate set in the image retrieval direction; The weight calculation module is used to calculate the final weight of the sample pair based on the first ranking similarity and the second ranking similarity, wherein the final weight is positively correlated with the semantic matching degree of the sample pair; The model update module is used to train the sample pairs in the candidate set according to the final weights and update the parameters of the cross-modal alignment model. The image-text matching module is used to input the data to be matched into the updated model to obtain cross-modal features, and obtain the image-text matching result corresponding to the data to be matched based on the similarity between the cross-modal features.

10. An electronic device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the cross-modal graph-text matching method based on order consistency as described in any one of claims 1 to 8.

Images