Image-text matching method based on local feature confidence

Abstract

According to the image-text matching method based on the local feature confidence coefficient, the threshold values of positive and negative samples are dynamically adjusted by performing mathematical modeling on the positive and negative samples, and the robustness and classification accuracy of the model are remarkably improved in combination with an attention mechanism and a sparse matrix; meanwhile, the calculation amount is reduced, the features can be efficiently extracted and collected, and the calculation efficiency is improved.

Description

Technical Field

[0001] This invention belongs to the field of cross-modal retrieval technology, specifically relating to an image-text matching method based on local feature confidence. Background Technology

[0002] In multimodal learning, calculating the similarity between images and text is a crucial task, widely applied in image retrieval, text retrieval, and image-text matching. Traditional similarity calculation methods typically rely on manual feature extraction and simple similarity metrics, such as cosine similarity. However, these methods fail to fully utilize the semantic information of images and text, resulting in low accuracy in similarity calculations. In recent years, deep learning methods have made significant progress in multimodal tasks, but existing methods still suffer from several limitations, such as insufficient modeling of local features and inadequate fusion of global semantic information. Summary of the Invention

[0003] To address the aforementioned problems, the purpose of this invention is to provide an image-text matching method based on local feature confidence, which solves the problems of insufficient modeling of local features and inadequate fusion of global semantic information in existing methods.

[0004] To achieve the above objectives, the technical solution adopted by the present invention includes:

[0005] An image-text matching method based on local feature confidence includes the following steps:

[0006] S1, retrieve the image and the corresponding text;

[0007] Extract features from n local regions in the image and embed them to obtain n local region vectors. Average the values ​​of all local region vectors to obtain the average region vector. Then, perform attention calculation on the average region vector to obtain the global image feature v. glo ;

[0008] Extract m word features from each text and embed them to obtain m word vectors. Calculate the average of all word vectors to obtain the average word vector. Then, perform attention calculation on the average word vector to obtain the global text feature u. glo ;

[0009] S2, using a self-attention mechanism, enhances the global representation of all local region vectors obtained in S1 and all word vectors of each text, resulting in n enhanced local region vectors v. i and m augmented word vectors u for each text j ;

[0010] S3, calculate each enhanced local region vector v for each text using equation (1).i Local similarity between the text context related to the enhanced local region vector We obtain a local similarity set, which includes n local similarities;

[0011]

[0012] Where W represents the learnable parameter matrix, v i This represents the vector of the i-th enhanced local region. Let α represent the text context corresponding to the feature of the i-th image region. i1 u1、α i2 u2、…、α ij u j The accumulation;

[0013] Among them, u j Let α represent the vector of the j-th augmented word. ij This represents the attention weights for the positive correlation between the i-th augmented local region vector and the j-th augmented word vector;

[0014] S4, calculate each enhanced word vector u using equation (4). j Word similarity between the visual context related to the augmented word vector Obtain the word similarity set, where each text's word similarity set includes m word similarities;

[0015]

[0016] in, Let α represent the visual context corresponding to the j-th augmented word vector. 1j v1、α 2j v2、…、α ij v j The accumulation;

[0017] S5, calculate the global image features v using equation (5). glo and text global features u glo global semantic similarity s glo ;

[0018]

[0019] S6, aggregate all the enhanced local region vectors obtained in S2 to obtain n region aggregated features. Each enhanced word vector is then compared with the region aggregation feature. Perform dot product matching to obtain the initial attention matrix H, which includes m×n original similarity scores between the enhanced local region vectors and the enhanced word vectors;

[0020] S7. Subtract the initial threshold from each feature in the initial attention matrix H obtained in S6 to obtain the first round attention matrix.

[0021] The first round of attention matrix is ​​masked with negative samples to obtain the first round of positive sample matrix; the first round of attention matrix is ​​masked with positive samples to obtain the first round of negative sample matrix.

[0022] Probabilistic modeling is performed on the first round of positive sample matrix and the first round of negative sample matrix to obtain the first round of optimized threshold;

[0023] S8, subtract the first-round optimized threshold obtained in S7 from each feature in the initial attention matrix obtained in S6 to obtain the second-round attention matrix;

[0024] The second-round attention matrix is ​​masked with negative samples to obtain the second-round positive sample matrix; the second-round attention matrix is ​​masked with positive samples to obtain the second-round negative sample matrix.

[0025] Probabilistic modeling is performed on the positive sample matrix and the negative sample matrix of the second round to obtain the optimized threshold for the second round.

[0026] S9 repeats S8 k times to obtain the positive sample matrix of the kth round and the optimized threshold t of the kth round. k ;

[0027] S10, calculate the regional aggregation characteristics of S6 using equation (6). The similarity ε between the positive sample matrix of round k and S9 k ;

[0028]

[0029] Where λ is the scaling parameter, H ij This represents the original similarity score between the i-th enhanced local region vector and the j-th enhanced word vector;

[0030] S11, calculate the confidence level C of this paper using equation (7), where C includes {c1, c2, ..., c i ,…,c n}, c i This represents the confidence level corresponding to the i-th image region;

[0031] C=σ(LayerNorm([ε1,ε2,…,ε k ])) (7)

[0032] S12, c i Local similarity obtained by multiplying by S3 Obtain the confidence scores corresponding to the n enhanced local region vectors;

[0033] Then, the global semantic similarity s obtained from S5 is used to... glo and the confidence level corresponding to the enhanced local region vector Summarized as

[0034] Then, the global semantic similarity s obtained from S5 is used to... glo Similarity to any word obtained from S4 Summarized as

[0035] S13, the S obtained from S12 v and S u A three-layer self-attention enhancement is performed through a self-attention inference module to extract and concatenate deep representations of each modality. Then, the representations are mapped to scalars through a linear layer and normalized by Sigmoid to obtain similarity scores. Based on the similarity scores, the matching results between the image and each text are determined.

[0036] Preferably, α ij Specifically as formula (2);

[0037]

[0038] Where γ represents the first scaling parameter, c ij [c] represents the original relevance score between the i-th augmented local region vector and the j-th augmented word vector. ij ] + c represents the corresponding positive sample ij Perform ReLU operation. [c] represents the normalized [c] ij ] + .

[0039] Preferably, γ = 20.

[0040] Preferably, k = 36.

[0041] Preferably, λ = 20.

[0042] Compared with the prior art, the advantages of the present invention are:

[0043] The present invention provides an image-text matching method based on local feature confidence. By mathematically modeling positive and negative samples separately to dynamically adjust the thresholds of positive and negative samples, and combining attention mechanism and sparse matrix, the robustness and classification accuracy of the model are significantly improved. At the same time, the amount of computation is reduced, and features can be extracted and collected efficiently, thus improving computational efficiency. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, this application will be described in further detail below. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application.

[0046] Example

[0047] This embodiment provides 1. an image-text matching method based on local feature confidence, characterized by the following steps:

[0048] S1, retrieve the image and the corresponding text;

[0049] Extract features from n local regions in the image and embed them to obtain n local region vectors. Average the values ​​of all local region vectors to obtain the average region vector. Then, perform attention calculation on the average region vector to obtain the global image feature v. glo ;

[0050] Extract m word features from each text and embed them to obtain m word vectors. Calculate the average of all word vectors to obtain the average word vector. Then, perform attention calculation on the average word vector to obtain the global text feature u. glo ;

[0051] This embodiment, given an image V, leverages the advantages of bottom-up attention by employing a Faster R-CNN object detector trained on the Visual Genomes dataset with a ResNet-101 backbone to extract 36 local region vectors from the image. These local region vectors are then linearly mapped to visual vectors in a common embedding space. The global features v of the image... glo The calculation is performed using an attention mechanism that uses the average value of regional features as the query key. For each text's word vector, a Bi-GRU (Bidirectional Gated Recurrent Unit) is used for extraction, and the global features u of each text are calculated. glo The calculation is performed using an attention mechanism that uses the average value of word features as the query key.

[0052] S2, using a self-attention mechanism, enhances the global representation of all local region vectors obtained in S1 and all word vectors of each text, resulting in 36 enhanced local region vectors v. i and m augmented word vectors u for each text j .

[0053] S3, calculate each enhanced local region vector v for each text using equation (1). i Local similarity between the text context related to the enhanced local region vector A local similarity set is obtained, wherein the local similarity set includes n local similarities;

[0054]

[0055] Where W represents the learnable parameter matrix, v i This represents the vector of the i-th enhanced local region. Let α represent the text context corresponding to the feature of the i-th image region. i1 u1、α i2 u2、…、α ij u j The accumulation;

[0056] Among them, u j Let α represent the vector of the j-th augmented word. ij This represents the attention weights for the positive correlation between the i-th augmented local region vector and the j-th augmented word vector.

[0057] S4, calculate each enhanced word vector u using equation (4). j Word similarity between the visual context related to the augmented word vector Obtain the word similarity set, where each text's word similarity set includes m word similarities;

[0058]

[0059] in, Let α represent the visual context corresponding to the j-th augmented word vector. 1j v1、α 2j v2、…、α ij v j The accumulation, α ij Specifically as formula (2);

[0060]

[0061] Where γ represents the first scaling parameter, γ = 20; c ij [c] represents the original relevance score between the i-th augmented local region vector and the j-th augmented word vector.ij ] + c represents the corresponding positive sample ij Perform ReLU operation. [c] represents the normalized [c] ij ] + .

[0062] S5, calculate the global image features v using equation (5). glo and text global features u glo global semantic similarity s glo ;

[0063]

[0064] S6, aggregate all the enhanced local region vectors obtained in S2 to obtain n region aggregated features. Each enhanced word vector is then aggregated with the region feature. Perform dot product matching to obtain the initial attention matrix H, which includes m×n original similarity scores between the enhanced local region vectors and the enhanced word vectors.

[0065] S7. Subtract the initial threshold from each feature in the initial attention matrix H obtained in S6 to obtain the first round attention matrix.

[0066] The first round attention matrix is ​​masked with negative samples to obtain the first round positive sample matrix; the first round attention matrix is ​​masked with positive samples to obtain the first round negative sample matrix; the first round positive sample matrix and the first round negative sample matrix are subjected to probability modeling to obtain the first round optimized threshold.

[0067] S8, subtract the first-round optimized threshold obtained in S7 from each feature in the initial attention matrix obtained in S6 to obtain the second-round attention matrix;

[0068] The second round attention matrix is ​​masked with negative samples to obtain the second round positive sample matrix; the second round attention matrix is ​​masked with positive samples to obtain the second round negative sample matrix; the second round positive sample matrix and the second round negative sample matrix are subjected to probability modeling to obtain the second round optimized threshold.

[0069] S9 repeats S8 k times to obtain the positive sample matrix of the kth round and the optimized threshold t of the kth round. k In this embodiment, k = 36.

[0070] S10, calculate the regional aggregation characteristics of S6 using equation (6). The similarity ε between the positive sample matrix of round k and S9 k ;

[0071]

[0072] Where λ is the scaling parameter, λ = 20; H ij This represents the original similarity score between the i-th enhanced local region vector and the j-th enhanced word vector.

[0073] S11, calculate the confidence level C of this paper using equation (7), where C includes {c1, c2, ..., c i ,…,c n}, c i This represents the confidence level corresponding to the i-th image region;

[0074] C=σ(LayerNorm([ε1,ε2,…,ε k ])) (7)

[0075] S12, c i Local similarity obtained by multiplying by S3 Obtain the confidence scores corresponding to the n enhanced local region vectors;

[0076] Then, the global semantic similarity s obtained from S5 is used to... glo and the confidence level corresponding to the enhanced local region vector Summarized as Then, the global semantic similarity s obtained from S5 is used to... glo Similarity to any word obtained from S4

[0077] S13, the S obtained from S12 v and S u A three-layer self-attention enhancement is performed through a self-attention inference module to extract and concatenate deep representations of each modality. Then, the representations are mapped to scalars through a linear layer and normalized by Sigmoid to obtain a similarity score. Based on the similarity score, the matching results between the image and each text are determined.

[0078] In this embodiment, all the obtained similarity scores are assembled into a matrix sims, where the element sims[i][j] represents the cross-modal similarity score between the i-th image and the j-th text description. Based on this matrix, the recall rate is calculated through bidirectional ranking statistics: the similarity of all matching items for each image (or text) is sorted in descending order, the best ranking position of its true counterpart is recorded, and the proportion of true matches in the top Q positions (Q = 1, 5, 10) is used as the Recall@K index. At the same time, the median and average ranking are calculated to comprehensively evaluate the retrieval performance.

[0079] The cross-modal matching model is optimized by comparing the distance between positive and negative sample pairs: First, the diagonal elements (positive sample pair scores) of the similarity matrix sims are extracted and expanded into comparison benchmark matrices d1 and d2. Then, the negative sample marginal loss of each text relative to all images and the negative sample marginal loss of each image relative to all texts are calculated separately, both using the ReLU activation marginal loss formula max(0, margin + negative sample score - positive sample score), with the margin set to 0.2. After masking the positive sample comparisons at the diagonal positions, the max_violation flag determines whether to retain all negative sample losses or only the loss of the hardest negative sample (maximum value). When set to True, only the loss value of the hardest negative sample corresponding to each query is retained; if set to False, the sum of the losses of all negative samples is calculated to balance the overall distribution. Finally, the sum of the text-image bidirectional losses is returned, forcing the model to bring positive sample pairs closer while pushing away negative samples, thereby enhancing the discriminative power of cross-modal matching.

[0080] As shown in Table 1, the model score in this embodiment is: Recall = 515.4, which is the best performance among the comparison models.

[0081] Table 1. Experimental Results of Flicker30k

[0082]

[0083] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

[0084] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0085] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process or method. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, or article that includes that element. In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not performed.

[0086] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image-text matching method based on local feature confidence, characterized in that, Specifically, the following steps are included: S1, retrieve the image and the corresponding text; Extract features from n local regions in the image and embed them to obtain n local region vectors. Average the values ​​of all local region vectors to obtain the average region vector. Then, perform attention calculation on the average region vector to obtain the global image feature v. glo ; Extract m word features from each text and embed them to obtain m word vectors. Calculate the average of all word vectors to obtain the average word vector. Then, perform attention calculation on the average word vector to obtain the global text feature u. glo ; S2, using a self-attention mechanism, enhances the global representation of all local region vectors obtained in S1 and all word vectors of each text, resulting in n enhanced local region vectors v. i and m augmented word vectors u for each text j ; S3, calculate each enhanced local region vector v for each text using equation (1). i Local similarity between the text context related to the enhanced local region vector A local similarity set is obtained, wherein the local similarity set includes n local similarities; Where W represents the learnable parameter matrix, v i This represents the vector of the i-th enhanced local region. Let α represent the text context corresponding to the feature of the i-th image region. i1 u1、α i2 u2、…、α ij u j The accumulation; Among them, u j Let α represent the vector of the j-th augmented word. ij This represents the attention weights for the positive correlation between the i-th augmented local region vector and the j-th augmented word vector; S4, calculate each enhanced word vector u using equation (4). j Word similarity between the visual context related to the augmented word vector Obtain the word similarity set, where each text's word similarity set includes m word similarities; in, Let α represent the visual context corresponding to the j-th augmented word vector. 1j v1、α 2j v2、…、α ij v j The accumulation; S5, calculate the global image features v using equation (5). glo and text global features u glo global semantic similarity s glo ; S6, aggregate all the enhanced local region vectors obtained in S2 to obtain n region aggregated features. Each enhanced word vector is then aggregated with the region feature. Perform dot product matching to obtain the initial attention matrix H, which includes m×n original similarity scores between the enhanced local region vectors and the enhanced word vectors; S7. Subtract the initial threshold from each feature in the initial attention matrix H obtained in S6 to obtain the first round attention matrix. The first round attention matrix is ​​masked with negative samples to obtain the first round positive sample matrix; the first round attention matrix is ​​masked with positive samples to obtain the first round negative sample matrix. Probabilistic modeling is performed on the first round of positive sample matrix and the first round of negative sample matrix to obtain the first round of optimized threshold; S8, subtract the first-round optimized threshold obtained in S7 from each feature in the initial attention matrix obtained in S6 to obtain the second-round attention matrix; The second round attention matrix is ​​masked with negative samples to obtain the second round positive sample matrix; the second round attention matrix is ​​masked with positive samples to obtain the second round negative sample matrix. Probabilistic modeling is performed on the positive sample matrix and the negative sample matrix of the second round to obtain the optimized threshold for the second round. S9 repeats S8 k times to obtain the positive sample matrix of the kth round and the optimized threshold t of the kth round. k ; S10, calculate the regional aggregation characteristics of S6 using equation (6). The similarity ε between the positive sample matrix of round k and S9 k ; Where λ is the scaling parameter, H ij This represents the original similarity score between the i-th enhanced local region vector and the j-th enhanced word vector; S11, calculate the confidence level C of this paper using equation (7), where C includes {c1, c2, ..., c i ,…,c n }, c i This represents the confidence level corresponding to the i-th image region; C=σ(Layernorm([ε1,ε2,…,ε k ])) (7) S12, c i Local similarity obtained by multiplying by S3 Obtain the confidence scores corresponding to the n enhanced local region vectors; Then, the global semantic similarity s obtained from S5 is used to... glo and the confidence level corresponding to the enhanced local region vector Summarized as Then, the global semantic similarity s obtained from S5 is used to... glo Similarity to any word obtained from S4 Summarized as S13, the S obtained from S12 v and S u A three-layer self-attention enhancement is performed through a self-attention inference module to extract and concatenate deep representations of each modality. Then, the representations are mapped to scalars through a linear layer and normalized by Sigmoid to obtain a similarity score. Based on the similarity score, the matching results between the image and each text are determined.

2. The image-text matching method based on local feature confidence as described in claim 1, characterized in that, α ij Specifically as formula (2); Where γ represents the first scaling parameter, c ij [c] represents the original relevance score between the i-th augmented local region vector and the j-th augmented word vector. ij ] + c represents the corresponding positive sample ij Perform ReLU operation. [c] represents the normalized [c] ij ] + .

3. The image-text matching method based on local feature confidence as described in claim 2, characterized in that, γ = 20.

4. The image-text matching method based on local feature confidence as described in claim 1, characterized in that, k＝36。 5. The image-text matching method based on local feature confidence as described in claim 1, characterized in that, λ = 20.

Images