Image-text cross-modal retrieval method and system

Abstract

The invention discloses an image-text cross-modal retrieval method, which comprises the following steps of: constructing an image-text data pair set, and respectively carrying out feature extraction on data of an image modal and a text modal; extracting hierarchical features of the image and the text through a multi-level attention distillation network, and performing inter-modal fine-grained alignment by adopting a cross attention mechanism and knowledge distillation to obtain fusion features; a dynamic batch clustering strategy is adopted to optimize feature alignment, a positive intra-class distance and a positive and negative inter-class distance are calculated, a distance correlation coefficient is adopted to calculate feature similarity, a joint loss function is constructed, and the loss function is minimized, intra-class distance correlation of similar samples is minimized, and inter-class distance correlation of heterogeneous samples is maximized; and mapping the fusion features to a public feature space, calculating the similarity between the fusion features and target modal features, and outputting image-text cross-modal and single-modal retrieval results. The technical problem that the retrieval precision and efficiency are limited due to the fact that modal heterogeneity and semantic heterogeneity are difficult to guarantee in an existing method is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and information retrieval technology, and in particular to a method and system for cross-modal retrieval of images and texts. Background Art

[0002] With the explosive growth of multimodal data, such as images, text, and videos, user retrieval needs have gradually shifted from single modality to cross-modality, for example, retrieving related images through text descriptions; however, the core challenge facing cross-modal retrieval is that the feature representations of data in different modalities are significantly heterogeneous, and semantic consistency is difficult to ensure, resulting in limited retrieval accuracy and efficiency.

[0003] Currently, existing methods mainly fall into the following four categories: Subspace mapping methods, such as CCA and DCCA, map multimodal data to a common subspace through linear or nonlinear projection. However, these methods rely on the global statistical correlation of paired samples and ignore local contextual information within the modality, resulting in insufficient fine-grained matching capabilities. Cross-modal hashing methods generate binary codes through hash functions to improve retrieval efficiency, but fixed-length hash codes have difficulty expressing hierarchical semantics. Deep learning methods, such as DSCMR and ACMR, extract discriminative features through end-to-end networks but fail to address the problem of insufficient inter-modal interaction. Generative methods, such as GANs and VAEs, compensate for modal differences through generative adversarial networks, but the generation direction is uncontrollable and the generated content has a high rate of repetition, resulting in semantic conflicts when generating images from text. Furthermore, existing contrastive losses only measure the similarity of joint distributions, ignoring differences in marginal distributions. The core issues of existing cross-modal retrieval technologies focus on modal heterogeneity, fine-grained alignment, and deep semantic modeling. Therefore, it is urgent to propose a cross-modal image and text retrieval method and system to solve the technical problems of existing methods that are difficult to ensure due to modal heterogeneity and semantic heterogeneity, resulting in limited retrieval accuracy and efficiency. Summary of the Invention

[0004] The main purpose of the present invention is to propose a cross-modal image-text retrieval method and system, aiming to solve the technical problem that the existing methods are difficult to ensure due to modal heterogeneity and semantic heterogeneity, resulting in limited retrieval accuracy and efficiency.

[0005] To achieve the above-mentioned object, the present invention provides a method for cross-modal image-text retrieval, wherein the method comprises the following steps:

[0006] S1. Constructing a set of image-text data pairs, wherein the image-text data pairs include data in an image modality and data in a corresponding text modality; and performing feature extraction on the data in the image modality and the data in the text modality respectively;

[0007] S2. Extract hierarchical features of images and texts through a multi-level attention distillation network, and use cross-attention mechanism and knowledge distillation to perform fine-grained alignment between modalities to obtain fused features;

[0008] S3. Use a dynamic batch clustering strategy to optimize feature alignment. Randomly select one category from each training batch as the positive category, and the remaining categories as negative categories. Calculate the intra-class distance of the positive category and the distance between positive and negative categories. Use the distance correlation coefficient to calculate feature similarity. Construct a joint loss function. By minimizing the loss function, minimize the intra-class distance correlation of similar samples and maximize the inter-class distance correlation of heterogeneous samples.

[0009] S4. Map the fused features to the common feature space, calculate the similarity with the target modality features, and output the cross-modal and single-modal retrieval results based on the similarity, that is, output the cross-modal and single-modal retrieval results of images and texts.

[0010] One of the preferred solutions is that in step S1, a VGGNet or AlexNet model is used to extract image features of the image modality data.

[0011] One of the preferred solutions is that in step S1, the BoW model is used to extract text features of the text modal data.

[0012] In one preferred solution, step S2 is specifically as follows:

[0013] S21, performing multi-level processing on the input image features and text features respectively through a multi-level attention distillation network; the multi-level attention distillation network includes an image feature learning network and a text feature learning network;

[0014] S22, generating a first image query vector, a first key vector, a first value vector, and a second text query vector, a second key vector, and a second value vector respectively according to the image features and text features of each level;

[0015] S23. Generate an inter-modal alignment weight vector by calculating a similarity score between the second value vector and the second key vector, and perform a weighted summation of the weight vector and each row of the first value vector to obtain a new image feature;

[0016] S24, performing linear transformation on the image features of each level to obtain a feature matrix, and performing linear transformation and normalization processing to generate weight coefficients, and obtaining the final image fusion feature through weighted summation;

[0017] S25. Symmetrically process the text features and image features after multi-level processing by the multi-layer attention distillation network to obtain the final text fusion features.

[0018] In one preferred solution, the image features are:

[0019]

[0020] in, is the image feature, i is the sample index, j is the level index, is the first value vector, is the first value vector The s-th row vector after reshaping, s is the number of rows, is the weight vector, is the second value vector With the first key vector The sth row after reshaping The similarity of is the second value vector, is the first key vector The mth row vector of .

[0021] In one preferred solution, the final image fusion feature is:

[0022] V i fuse =W v F i v

[0023] Among them, V i fuse is the final image fusion feature, W v is the weight coefficient, F i v is a characteristic matrix; f fusel () is the image feature linear layer, They are the image features corresponding to levels 1, 2, and 3 respectively. is the parameter corresponding to the linear layer of image features.

[0024] In one preferred solution, the intra-class distance of the positive class is calculated in step S3, specifically:

[0025] Calculate the mean vector of the positive class; specifically: calculate the mean vector of the image features respectively and the text feature mean vector And the image feature mean vector It is a positive category center;

[0026] Calculate the mean vector of image features The dual-centered distance matrix Double-neutralized distance matrix of image features And the double-neutralized distance matrix of text features

[0027] Calculate the intra-class distance correlation to obtain the intra-class distance of the positive class. Use the intra-class distance to quantify the average correlation between the image and text features within the positive class and the positive class center. The intra-class distance is:

[0028]

[0029] in, is the intra-class distance, m1 is the number of positive image-text pairs in a batch, and dCorr() represents the linear and nonlinear dependencies between features. is the mean vector of image features, is the image feature of the positive class, is the text feature of the positive class, k is the category index, i is the sample index, and j is the level index.

[0030] In one preferred solution, the distance between positive and negative classes is:

[0031]

[0032] in, is the distance between positive and negative classes, m2 is the amount of negative class image-text pairs in a batch, dCorr() represents the linear and nonlinear dependency between features, is the mean vector of image features, is the image feature of the negative class, is the text feature of the negative class, M is a set of categories in a batch, k is the category index, i is the sample index, and j is the level index.

[0033] In one preferred solution, step S3 constructs a joint loss function, which minimizes the intra-class distance correlation of samples of the same class and maximizes the inter-class distance correlation of samples of different classes by minimizing the loss function, specifically:

[0034] Construct the j-th layer clustering loss function about the image center And the text features are processed symmetrically, with the text feature mean vector is the positive class center, and the clustering loss function of the jth layer about the text center is obtained The clustering loss of the jth layer is L j ;

[0035] Calculate the clustering loss of the fusion feature, and get the total clustering loss L according to the clustering loss of each layer and the clustering loss of the fusion feature cluster ;

[0036] Based on the total clustering loss L cluster , construct a total loss optimization model, and by minimizing the loss function, minimize the intra-class distance correlation of similar samples and maximize the inter-class distance correlation of heterogeneous samples.

[0037] A system including the above-mentioned cross-modal image-text retrieval method comprises: a data input and feature extraction module, a feature interaction enhancement module, a semantic optimization module, and a retrieval execution module connected in sequence;

[0038] The data input and feature extraction module is used to receive image data and text data, and extract image features and text features of image-text data pairs;

[0039] The feature interaction enhancement module is used to extract hierarchical features of image features and text features and perform modality alignment;

[0040] The semantic optimization module is used to optimize feature alignment through a dynamic batch clustering strategy and perform loss calculation. By minimizing the loss function, the intra-class distance correlation of similar samples is minimized and the inter-class distance correlation of heterogeneous samples is maximized.

[0041] The retrieval execution module is used for cross-modal retrieval of images and texts and single-modal retrieval.

[0042] In the above technical solution of the present invention, the image-text cross-modal retrieval method includes the following steps: constructing a set of image-text data pairs, wherein the image-text data pairs include image modality data and corresponding text modality data; performing feature extraction on the image modality and text modality data respectively; extracting hierarchical features of images and texts through a multi-level attention distillation network, and using a cross-attention mechanism and knowledge distillation to perform fine-grained alignment between modalities to obtain fused features; using a dynamic batch clustering strategy to optimize feature alignment, randomly selecting a category from each training batch as the positive class, and the remaining categories as negative classes, calculating the intra-class distance of the positive class and the distance between positive and negative classes, and using the distance correlation coefficient to calculate feature similarity, constructing a joint loss function, and by minimizing the loss function, minimizing the intra-class distance correlation of similar samples and maximizing the inter-class distance correlation of heterogeneous samples; mapping the fused features to a common feature space, calculating the similarity with the target modality features, and outputting cross-modal and unimodal retrieval results based on the similarity, that is, outputting cross-modal and unimodal retrieval results of images and texts. The present invention solves the technical problem that the existing methods are difficult to ensure due to modal heterogeneity and semantic heterogeneity, resulting in limited retrieval accuracy and efficiency.

[0043] In the present invention, in order to learn the relationship between data of different modalities, the rectification and attention mechanisms are connected to learn the global feature correspondence between images and texts and the fine-grained contextual local information within images and texts. The semantic information in the original data is used to perform cross-modal fusion of image and text modal features, which not only improves the retrieval accuracy of cross-modal retrieval tasks, but also improves the retrieval accuracy in single-modal retrieval tasks, especially in small sample scenarios.

[0044] In the present invention, distance correlation is used as the similarity metric, which has more powerful feature discrimination ability than traditional Euclidean distance or cosine similarity; by calculating the total clustering loss composed of clustering losses at each level in the common representation space, the prediction loss composed of the Euclidean distance between the predicted label and the true label, and in order to prevent overfitting, the parameter L2 regularization term is given different weights to obtain the final loss function of the retrieval model, and by minimizing the loss function to optimize the model parameters, the optimized retrieval model is obtained. Since the global and local information of different modal data is fully utilized, the learning of fine-grained features between different modalities is strengthened, and the semantic discrimination and consistency are enhanced, the present invention can perform cross-modal retrieval well and obtain competitive results on multiple databases. At the same time, the joint optimization of dynamic clustering and distance correlation alleviates the problem of strong and weak modal deviation.

[0045] In the present invention, BDCPL pooling is used when calculating the loss function. BDCPL pooling enhances the discriminability of the common representation space through delinearization and global dependency modeling. Hierarchical distillation can not only transfer high-order semantics, enhance cross-modal alignment capabilities, but also compress the number of parameters. The present invention improves the accuracy and practicality of cross-modal retrieval through BDCPL pooling and hierarchical distillation, reduces computational costs, and achieves a balance between accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0047] Figure 1 This is a first schematic diagram of a method for cross-modal image and text retrieval according to an embodiment of the present invention;

[0048] Figure 2 This is a second schematic diagram of a method for cross-modal image and text retrieval according to an embodiment of the present invention;

[0049] Figure 3 Schematic diagram of the attention distillation network according to an embodiment of the present invention;

[0050] Figure 4 Schematic diagram of attention extraction in the attention distillation network according to an embodiment of the present invention;

[0051] Figure 5 Schematic diagram of attention fusion in the attention distillation network according to an embodiment of the present invention;

[0052] Figure 6This is a scatter plot of 2D data distribution in the UCIpendigits dataset according to an embodiment of the present invention;

[0053] Figure 7 This is a 2D scatter plot when the batch size is 32 in Experiment 1 of the embodiment of the present invention;

[0054] Figure 8 This is a 2D scatter plot when the batch size is 32 in Experiment 3 of the embodiment of the present invention;

[0055] Figure 9 This is a 2D scatter plot of the experiment 1 of the embodiment of the present invention when the batch size is 100;

[0056] Figure 10 This is a 2D scatter plot when the batch size is 100 in Experiment 3 of Example 3 of the present invention.

[0057] The realization of the objectives, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0059] It should be noted that all directional indications (such as up, down, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0060] In addition, the terms "first," "second," and so on, used in this disclosure are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include at least one of these features.

[0061] Moreover, the technical solutions between the various embodiments of the present invention may be combined with each other, but this must be based on the fact that ordinary technicians in this field can implement them. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0062] See also Figures 1-10According to one aspect of the present invention, the present invention provides a method for cross-modal image-text retrieval, wherein the method comprises the following steps:

[0063] S1. Constructing a set of image-text data pairs, wherein the image-text data pairs include data in an image modality and data in a corresponding text modality; and performing feature extraction on the data in the image modality and the data in the text modality respectively;

[0064] S2. Extract hierarchical features of images and texts through a multi-level attention distillation network, and use cross-attention mechanism and knowledge distillation to perform fine-grained alignment between modalities to obtain fused features;

[0065] S3. Use a dynamic batch clustering strategy to optimize feature alignment. Randomly select one category from each training batch as the positive category, and the remaining categories as negative categories. Calculate the intra-class distance of the positive category and the distance between positive and negative categories. Use the distance correlation coefficient to calculate feature similarity. Construct a joint loss function. By minimizing the loss function, minimize the intra-class distance correlation of similar samples and maximize the inter-class distance correlation of heterogeneous samples.

[0066] S4. Map the fused features to the common feature space, calculate the similarity with the target modality features, and output the cross-modal and single-modal retrieval results based on the similarity, that is, output the cross-modal and single-modal retrieval results of images and texts.

[0067] Specifically, in this embodiment, the VGGNet or AlexNet model is used in step S1 to extract the image features of the image modality data; the BoW model is used to extract the text features of the text modality data; the label vector y i One-hot encoding is used to ensure feature dimension alignment, where y i =(y i1 、y i2 ,...,y ic ), c is the total number of categories, i is the sample index, which represents the observation data of the i-th picture-text, that is, the i-th sample.

[0068] Specifically, in this embodiment, the present invention adopts the Pascal Sentence dataset, which contains 1000 pairs of image-text data, divided into 800 pairs of image-text datasets as training sets, 100 pairs of image-text datasets as validation sets, and 100 pairs of image-text data as test sets; a total of 20 category labels, covering a wide range of application scenarios, verifying the alignment ability of the model, and using the pre-trained VGGNet model to extract a 4096-dimensional image feature vector v for each image data i ∈R 4096 , use the BoW model to extract a 300-dimensional text feature vector t for each text datai ∈R 300 , the batch size is set to n = 64, and each pair of data is associated with 20 categories of semantic labels y i =(y i1 、y i2 ,...,y i20 ), using one-hot encoding.

[0069] Specifically, in this embodiment, step S2 is as follows:

[0070] S21, perform multi-level processing on the input image features and text features respectively through a multi-level attention distillation network; the multi-level attention distillation network includes an image feature learning network and a text feature learning network; each level of the network includes a feature transformation layer and a residual connection, and outputs the image features of each level and text features Wherein, j=1, 2, 3, j is the level index, that is, the number of levels; the residual connection is used to add the current level output to the original input or the previous level feature, retaining the original information and enhancing the feature stability;

[0071] S22, according to the image features of each level and text features Generate the first image query vector respectively First key vector First value vector and the second text query vector Second key vector and the second value vector

[0072] S23, by calculating the second value vector and the second key vector Similarity score, generate inter-modal alignment weight vector The weight vector With the first value vector The weighted sum of each row of is used to obtain the new image features, which is recorded as The processing of text features is symmetrical to that of image features; the weight vector of level j is The weight vector of m rows for:

[0073]

[0074] in, is the weight vector of the mth row of the jth level weight vector, is the key-value vector of the mth row of the first key-value vector at level j;

[0075] The weight vector for:

[0076]

[0077] in, is the second value vector With the first key vector The sth row after reshaping similarity;

[0078] The image features are:

[0079]

[0080] in, is the first value vector The s-th row vector after reshaping;

[0081] S24, linearly transform the image features at each level to obtain a feature matrix, and perform linear transformation and softmax normalization processing to generate weight coefficients, and obtain the final image fusion feature through weighted summation; the feature matrix is:

[0082]

[0083] Among them, F i v is the characteristic moment, f fusel is the image feature linear layer, They are the first, second and third level image features respectively. is the parameter corresponding to the linear layer of image features;

[0084] The weight coefficient is:

[0085]

[0086] Among them, W v is the weight coefficient of the image feature, The attention weights corresponding to the first, second, and third level image features respectively;

[0087] The final image fusion feature is:

[0088] V i fuse =W v F i v

[0089] Among them, V i fuse is the final image fusion feature, W v is the weight coefficient, F i v is a characteristic matrix; These are the image features corresponding to levels 1, 2, and 3 respectively;

[0090] S25, symmetrically process the text features and image features after multi-level processing of the multi-layer attention distillation network to obtain the final text fusion feature T i fuse .

[0091] Specifically, in this embodiment, see Figure 3-Figure 4 The present invention integrates the attention mechanism and distillation structure in the cross-modal system through the idea of multi-head attention and knowledge distillation. By constructing a knowledge distillation attention network, the hierarchical output rectified features are fused through the cross-attention mechanism to obtain new hierarchical features. Finally, all hierarchical features are fused through the self-attention mechanism to obtain the fusion features required for retrieval. The image feature learning network f(θ 11 θ 12 θ 13 θ α ), text feature learning network f(θ 21 θ 22 θ 23 θ α ); Each feature learning network consists of two sets of residual blocks and a set of MLP layers. For each residual block, a batch normalization layer is followed by the input. The multi-layer perceptron MLP consists of a linear layer, a regularization layer, and an activation layer. Finally, a simple fully connected layer is used to connect the two feature learners to the same classifier, where the number of labels of the classifier is determined by the number of labels in the dataset. The corresponding residual parameters of the two feature learning networks are defined as θ 1i' ,θ 2i' ; The image MLP layer parameters and text MLP layer parameters are defined as θ α ,θ α , the fully connected parameter of the classifier is defined as θ α ; Where i'=1,2, i' is the parameter corresponding to the i'th residual block, with θ 1i' For example, when i'=1, it represents the corresponding parameters of the first residual block of the image feature learning network;

[0092] Taking picture features as an example, the picture feature vector v i After the first layer residual block of the image feature learning network, the new features are output as The image features after residual connection Input the second residual block of the image feature learning network and output new features The image features after residual connection Input image feature learning network MLP layer and output new features Similarly, the text feature vector t i After the first layer residual block of the text feature learning network, the new features output are The text features after residual connection Input the second layer residual block of the text feature learning network and output new features The text features after residual connection Input text feature learning network MLP layer and output new features

[0093] See also Figure 4 , the linear extraction layers of the value V, key K and query Q corresponding to the j-th level of the image feature attention extraction part are The j-th layer image feature is obtained by linear extraction layer to obtain vector and is a d-dimensional vector, Reshape into s rows and d / s columns matrices respectively, and after reshaping, each row and The dimensions are the same, reshaping can learn the attention weights of different local information of features, where s and d can respectively adjust the number of rows and the output dimension of the linear layer; calculation and key The similarity of each row in Column concatenation to form a score vector The score vector and value The weighted sum of each row of is used to obtain the new image features, which are still recorded as

[0094] The linear extraction layers of the value V, key K and query Q corresponding to the j-th level of the text feature attention extraction part are The j-th layer text feature is extracted by the linear layer to obtain the vector and Will and Reshape into s rows and d columns matrices respectively, which can learn the attention weights of different local information of features, where s and d can respectively control the number of rows and the output dimension of the linear layer; calculate and key The similarity of each row in Column concatenation to form a score vector The score vector and value The weighted sum of each row of is used to obtain the new text feature, which is still recorded as

[0095] Query vector for the first image With the second key vector The mth row after reshaping The similarity is:

[0096]

[0097] in, is the second key vector The mth row vector after reshaping;

[0098] The weight vector of the text feature for:

[0099]

[0100] in, is the first image query vector With the second key vector The sth row after reshaping similarity;

[0101] The text features are:

[0102]

[0103] in, is the second text query vector The s-th row vector of ;

[0104] See also Figure 5 , the new features after attention extraction at each level Entering the same linear layer in sequence is Obtainable characteristic matrix Then enter the same weighted linear layer in turn Through softmax normalization, the image fusion feature V i fuse =W v F i v ; Similarly, for the text features extracted by attention at each level Symmetrically processed with the image features, the final text fusion feature T is obtained i fuse ;

[0105]

[0106] and are the label prediction values of the kth image feature and text feature respectively; where h() is the classifier function for classification, θ αis the fully connected parameter of the classifier, It is the k-th image feature and the k-th text feature after the third layer of attention distillation, k = 1, 2, ..., N.

[0107] Specifically, in this implementation, the intra-class distance of the positive class is calculated in step S3 as follows:

[0108] Calculate the mean vector of the positive class; specifically: calculate the mean vector of the image features respectively and the text feature mean vector And the image feature mean vector It is a positive category center;

[0109] Calculate the mean vector of image features The dual-centered distance matrix Image features of the positive class Double-neutralization distance matrix And the text features of the positive class Double-neutralization distance matrix The dual-centered matrix can reflect the global structure of the feature distribution and is used for the subsequent calculation of the distance covariance dCov. The distance covariance dCov measures the difference between the joint distribution of the two modal features and their respective marginal distributions, capturing the linear and nonlinear correlation between the two.

[0110] Calculate the intra-class distance correlation to obtain the intra-class distance of the positive class. Use the intra-class distance to quantify the average correlation between the image and text features within the positive class and the center of the positive class. Use the intra-class distance to quantify the average correlation between the image and text features within the positive class and the center of the class. The smaller the value, the more similar the features of the same class are. The intra-class distance is:

[0111]

[0112] in, is the intra-class distance, a batch category set refers to a set of data samples processed simultaneously in one data processing, m1 is the amount of positive class image-text pairs in a batch, dCorr( ) represents the linear and nonlinear dependency between features, is the mean vector of image features, is the image feature of the positive class, is the text feature of the positive class, k is the category index, i is the sample index, j is the level index, the category set in a batch is M, the positive category k∈M, and the categories except k in M are all negative categories, denoted as M\k;

[0113]

[0114] in, After standardization is the distance covariance between the positive class center and the positive class image feature, is the distance variance of the positive class center, is the distance variance of the positive image features;

[0115]

[0116] in, After standardization is the distance covariance between the positive class center and the positive class text feature, is the distance variance of the positive class center, is the distance variance of the positive text feature;

[0117]

[0118] Where s is The number of rows after splitting, r is the rth row, t is the tth column, Image features for the positive class Double neutralization distance matrix;

[0119]

[0120] in, Text features for the positive class Double neutralization distance matrix;

[0121]

[0122] in, is the mean feature vector of the positive image The dual-centered distance matrix The element in the rth row and tth column of ;

[0123]

[0124] The present invention aims to separate the positive class k from a batch, away from all the points of the negative class. It does not matter whether the negative class is clustered. Therefore, the intra-class distance for the negative class defaults to 0. On the other hand, if the negative class is further clustered, the points represented by two different labels in the negative class will be further mixed together, which is not conducive to our subsequent iterative separation of each label in the batch as a positive label.

[0125] Specifically, in this embodiment, for the calculation of the inter-class distance, since it is pre-set that the negative class is well clustered, each point in the negative class is the cluster center of the negative class, that is, the mean vector. Therefore, the inter-class distance is obtained by calculating the distance between the mean vector of the positive class and each point of the negative class, summing them, and finally calculating the mean. use Quantify the average distance between the center of the positive class and all negative class samples. The larger the value, the more separated the heterogeneous features are. The distance between positive and negative classes is:

[0126]

[0127] in, is the distance between positive and negative classes, the class set in a batch is M, k is the positive class, k∈M, and all classes except k in M are negative classes, recorded as M=M\k, m2 is the data volume of negative class image-text pairs in a batch, dCorr() represents the linear and nonlinear dependency between features, is the mean vector of image features, is the image feature of the negative class, is the text feature of the negative class, M is the category set in a batch, k is the category index, i is the sample index, and j is the level index;

[0128]

[0129] in, After standardization is the distance covariance between the positive class center and the negative class image feature, is the distance variance of the positive class center, is the distance variance of the negative image features;

[0130]

[0131] in, After standardization is the distance covariance between the positive center and the negative text feature, is the distance variance of negative text features;

[0132]

[0133] in, Text features for negative classes The dual-centered distance matrix;

[0134]

[0135] in, Image features for the negative class The dual-centered distance matrix;

[0136]

[0137] in, is the mean vector of positive image features Dual-centered distance matrix The element at row r and column t;

[0138]

[0139] Specifically, in this embodiment, step S3 constructs a joint loss function, which minimizes the intra-class distance correlation of samples of the same class and maximizes the inter-class distance correlation of samples of different classes by minimizing the loss function, specifically:

[0140] Construct the j-th layer clustering loss function about the image center And the text features are processed symmetrically, with the text feature mean vector is the positive class center, and the clustering loss function of the jth layer about the text center is obtained The clustering loss of the jth layer is The clustering loss function is:

[0141]

[0142] Among them, k is the positive category and M is the negative category;

[0143] Calculate the clustering loss of the fusion feature, and obtain the total clustering loss L according to the clustering loss of each layer and the clustering loss L4 of the fusion feature. cluster ;L cluster =L1+L2+L3+L4; where L1, L2, and L3 are the total clustering losses of the first, second, and third levels respectively;

[0144] Based on the total clustering loss L cluster , build a total loss optimization model, by minimizing the loss function, minimize the intra-class distance correlation of similar samples, and maximize the inter-class distance correlation of heterogeneous samples; the total loss function of the model includes clustering loss, prediction loss and L2 regularization term; clustering loss is the sum of the total clustering loss of each level, prediction loss is the Euclidean distance between the predicted label and the true label, and the L2 regularization term is to constrain network parameters to prevent overfitting;

[0145]

[0146] Among them, L pre is the label prediction loss, m is the number of samples in a batch, m=m1+m2, pre iv is the image modality prediction label of the i-th sample, Y is the true label of the i-th sample, it Predict the text modality label for the i-th sample;

[0147]

[0148] Among them, L is the joint loss function, that is, the loss function of the total loss optimization model, L clusteris the dynamic clustering loss, L pre is the label prediction loss, λ is the hyperparameter that controls the regularization strength, and θ is the set of parameters to be learned and optimized;

[0149] Specifically, in this embodiment, the fused features are mapped to the common feature space, the dCorr similarity with the target modality features is calculated, and the cross-modal and single-modal retrieval results are output based on the similarity, that is, the cross-modal and single-modal retrieval results of images and texts are output.

[0150] Specifically, in this embodiment, the present invention adopts the Pascal Sentence data set, the batch size is set to n=64, batch is the batch size, and a dynamic clustering strategy is adopted. In each training batch, a category is randomly selected as the positive class, the sample corresponding to the positive class is the positive class sample, and the remaining categories are the negative class samples. The positive class contains m1 pairs of image-text samples, and the negative class contains m2 pairs of samples; the distance correlation calculation cannot directly calculate the correlation between a single image feature vector and a single text feature vector. Therefore, the positive class image feature and text feature data are split from one row into s rows, that is, the positive class image feature and the text features of the positive class By dimension and Become a dimension and The default calculation of loss function is as well as The image and text features of the positive and negative classes after the i-th data split at the j-th level;

[0151] For each level, taking the jth level as an example, first, calculate the mean vector of the positive class and calculate the mean vector of the image features respectively. and the text feature mean vector First, take the image feature mean vector is the center of the positive class, and the mean vector in the positive class is calculated The dual-centered distance matrix For the mean vector For all rows, calculate the Euclidean matrix between each two to obtain the symmetric matrix D X ∈R s×s , in, is the mean vector The rth row vector of ; calculate the row mean Column mean and the global mean Get the dual-centered distance matrix Similarly, calculate the image features of the positive class Double-neutralization distance matrix And the text features of the positive class Double-neutralization distance matrix Then calculate the distance correlation, and finally get the intra-class distance of the positive class The negative class does not need to be forced to cluster, it just needs to stay away from the entire class;

[0152] If the intra-class distance of the negative class is reduced, negative samples of different labels will be mixed, affecting the subsequent iterative separation. Therefore, the default intra-class distance of the negative class is 0, and only the distance between the positive and negative classes is optimized; since the default negative class is well clustered, each point in the negative class is the cluster center of the negative class, that is, the mean vector. Therefore, the distance between the mean vector of the positive class and each point of the negative class is calculated, summed, and finally the mean is calculated to obtain the inter-class distance. In the same way, calculate the image features of the negative class The dual-centered distance matrix and text features The dual-centered distance matrix Then calculate the distance between classes Construct the clustering loss function of the jth level about the image center Symmetrical processing of text features can obtain the clustering loss function of the jth level about the text center: The total clustering loss at the jth level is For the fusion features, the same calculation is performed to obtain the clustering loss of the fusion features, which is recorded as L4, and the total clustering loss is L cluster =L1+L2+L3+L4; The model is optimized by total loss, with the goal of minimizing the intra-class distance of the positive class and maximizing the distance between the positive and negative classes, while not constraining the internal structure of the negative class. The total loss function of the model includes clustering loss, prediction loss, and L2 regularization term; the total clustering loss is the sum of the losses of each layer; the prediction loss is the Euclidean distance between the predicted label and the true label; the L2 regularization term constrains the network parameters to prevent overfitting; for the transmembrane state retrieval task, given a query, such as image v q , extract its fusion features For all text features T in the database i fuse , calculation and The distance correlation is sorted in descending order by distance correlation, and the top K most relevant cross-modal samples are returned; since the model regards images and texts as data in the same space, that is, it does not distinguish between modalities, it can be directly used for single-modal retrieval; for single-modal retrieval tasks, given a query, such as image v q , extract its fusion features For all image features V in the database i fuse Calculate the distance correlation, sort by distance correlation in descending order, and return the top K most relevant cross-modal samples.

[0153] Specifically, in this embodiment, in order to better demonstrate the retrieval performance of the model for small sample data, medium sample data, and large sample data, in addition to the Pascal Sentence dataset, the small sample dataset Wikipedia, the medium sample dataset NUS-WIDE-10k, and the large sample dataset MS COCO dataset were selected according to the data size, and the retrieval performance was evaluated using the Recall@K and MAP@K indicators; the four datasets were independently trained to verify the universality of the model under different data scales; during the specific training, the initial learning rate was set to 0.05, the momentum was 0.9, and the SGD optimizer with a decay of 0.0005 was used to optimize the model. The batch size for small data was 64, the batch size for medium sample data was 128, and the batch size for large data was 512, and 100 epochs were used;

[0154] The Wikipedia dataset contains 2866 pairs of image-text data with 10 category labels, 2173 pairs of image-text data are divided into training sets, 200 pairs of image-text data are divided into validation sets, and 493 pairs of image-text data are divided into test sets. VGGNeT is used to extract 4096-dimensional features for each image. i ∈R 4096 , use BoW to extract 5000-dimensional features for each text i ∈R 5000 ; Each pair of data is associated with 10 types of semantic labels y i =(y i1 ,y i2 ,…,y i10 ), one-hot encoding is used; the batch size is set to m=64;

[0155] The NUS-WIDE-10k dataset contains 10,000 pairs of image-text data with 10 category labels. 8,000 pairs of image-text data are divided into training sets, 1,000 pairs of image-text data are divided into validation sets, and 1,000 pairs of image-text data are divided into test sets. VGGNeT is used to extract 4096-dimensional features for each image. i ∈R 4096 , use BoW to extract 1000-dimensional features for each text i ∈R 1000 ; Each pair of data is associated with 10 types of semantic labels y i =(y i1 ,y i2 ,…,y i10 ), using one-hot encoding; the batch size is set to m = 128;

[0156] The MS COCO dataset contains 123,287 pairs of image-text data, with a total of 80 category labels. 113,287 pairs of image-text data are divided into a training set, 5,000 pairs of image-text data are divided into a validation set, and 5,000 pairs of image-text data are divided into a test set. AlexNet is used to extract 4096-dimensional features for each image. i ∈R 4096 , use BoW to extract 2000-dimensional features for each text i ∈R 2000 ; Each pair of data is associated with 80 categories of semantic labels y i =(y i1 ,y i2 ,…,y i80 ), one-hot encoding is used; the batch size is set to m=512;

[0157] In addition to the above, the training process of the model on the small sample dataset Wikipedia, the medium sample dataset NUS-WIDE-10k, and the large sample dataset MS COCO dataset is consistent with that of the Pascal Sentence dataset.

[0158] Specifically, in this embodiment, a dynamic batch clustering strategy comparison experiment is conducted on the present invention. The pengitigs dataset is one of the commonly used handwritten digital datasets, which contains feature vectors and corresponding classification labels of 10,992 handwritten digital images. Each digital image consists of 16 digital features, of which 7,494 are divided into training sets and 3,498 are divided into test sets. Three experiments are conducted using this dataset: the first experiment divides the data of the existing labels in the batch, calculates the intra-group distance and inter-group distance of each category, and constructs a loss function based on the ratio; the second experiment iteratively selects a category data as the data in the batch The data of the other categories are all regarded as negative categories. The sum of the intra-group distance between the positive and negative categories and the sum of the inter-group distance between the positive and negative categories are calculated, and the ratio is used to construct the loss function. The third experiment iteratively selects a category data as the positive category for the data in the batch, and the rest of the category data are all regarded as negative categories. The intra-group distance of the positive category is calculated, and the intra-group distance of the negative category is defaulted to 0. The sum of the inter-group distance between the positive and negative categories is calculated, and the ratio is used to construct the loss function. The experimental results are verified from two aspects: clustering accuracy and actual cluster scatter plot. The experiment is conducted on a simple three-layer linear layer network. The number of nodes in the feature extraction layer are 16, 32, and 16 respectively. The batch size is 32 and 100 respectively. The metric is MAP, see Figure 6 , which is the 2D scatter plot of the original data set, see Figure 8-Figure 9 They are the 2D scatter plots of Experiment 1 and Experiment 3 respectively; First, by observing Figure 6The degree of confusion of the pengitigs dataset points can be seen, and the categories overlap with each other. As shown in Table 1, the MAP value of Experiment 2 is much lower than that of Experiment 1 and Experiment 3, proving the effectiveness of not further reducing the intra-class distance for the negative class within the batch. Compared with Experiment 1, the MAP of Experiment 3 is basically the same under the condition of batch size = 100, and increases by nearly 3 percentage points under the condition of batch size = 32, thus confirming the effectiveness of dividing the data into positive and negative classes within the batch. Figure 7 and Figure 9 While Experiment 1 successfully separated points from different categories, some clusters within the same category were very loose and lacked focus. Experiment 3, on the other hand, not only separated points from different categories but also strengthened the clustering of points within the same category, consistent with the principle of clustering points within the same category and separating different categories. The dynamic batch clustering strategy exhibits a performance advantage over the fixed batch strategy in cross-modal retrieval. We also found that smaller batch sizes yielded better results, consistent with the common practice of using small batch sizes in visual processing to improve the learning of details.

[0159] Table 1 shows the MAP metric of different group classifications on the pengitigs dataset

[0160]

[0161] According to another aspect of the present invention, the present invention provides a cross-modal image-text retrieval system, comprising: a data input and feature extraction module, a feature interaction enhancement module, a semantic optimization module, and a retrieval execution module connected in sequence;

[0162] The data input and feature extraction module is used to receive image data and text data, and extract image features and text features of image-text data pairs;

[0163] The feature interaction enhancement module is used to extract hierarchical features of image features and text features and perform modality alignment;

[0164] The semantic optimization module is used to optimize feature alignment through a dynamic batch clustering strategy and perform loss calculation. By minimizing the loss function, the intra-class distance correlation of similar samples is minimized and the inter-class distance correlation of heterogeneous samples is maximized.

[0165] The retrieval execution module is used for cross-modal retrieval of images and texts and single-modal retrieval.

[0166] Specifically, in this embodiment, the system also includes an input interface configured to receive a set of image-text data pairs, where the image-text data pairs include data in the image modality and data in the corresponding text modality. The data input and feature extraction modules perform feature extraction on the image modality and text modality data respectively, and output image features and text features of the image-text data pairs. At the same time, each image-text data pair has a corresponding semantic label vector to provide guidance for subsequent supervised learning. After feature extraction is completed, hierarchical feature distillation and attention alignment are performed. The feature interaction enhancement module inputs the extracted features into the three-level attention distillation network for step-by-step processing. In the first-level processing, the image and text features are respectively subjected to preliminary feature transformation through the residual block with the BN layer, and the output dimension is and The first-level output features are fused with the original input features through residual connections to form the input features of the second-level network. The query vector, key vector and value vector are generated through the cross-attention mechanism for the features after residual connection. The inter-modal alignment weight is calculated based on the cosine similarity between the query vector and the key vector, and the value vector is weighted summed to achieve preliminary alignment of image and text features. At the same time, the original feature information is retained through residual connections to enhance the alignment stability. The third-level processing fuses the features of the first two levels and then undergoes dimensionality reduction through the MLP layer. The final output dimension is and Refined features; the multi-level structure of the feature interaction enhancement module effectively realizes the gradual purification of features and modality alignment; in the model optimization stage, the system innovatively adopts a dynamic clustering strategy; in each training batch, a category is randomly selected as the positive sample, and the remaining categories are used as negative samples. The positive class contains m1 pairs of image-text samples, and the negative class contains m2 pairs of samples; innovatively adopts distance correlation as the similarity metric. Distance correlation can measure correlation from both linear and nonlinear aspects. Distance correlation can directly measure the degree of correlation between two variables of different dimensions, eliminating the problem of dimensionality difference and linear correlation assumption between modal data; distance correlation metric is used to replace traditional Euclidean distance. Distance correlation metric is calculated through a dual-centered matrix, which can simultaneously consider the joint and marginal distribution information of features, and simultaneously optimize the joint distribution and marginal distribution. Compared with traditional Euclidean distance or cosine similarity, it has more powerful feature discrimination ability; the total loss function of the model includes clustering loss, prediction loss and L2 regularization term; the total clustering loss is the sum of the losses of each level; the prediction loss is the difference between the predicted label and the true label Euclidean distance; L2 regularization is to constrain network parameters to prevent overfitting; by minimizing the intra-class distance correlation loss of positive samples, similar features are clustered together in a common semantic space; at the same time, minimizing the inter-class distance correlation loss between positive and negative samples, different categories of features are separated from each other, so as to achieve the purpose of clustering each type of data together and separating different types of data; after training, the model enters the application stage; when an image query is received, the system converts the query image into a feature vector, performs similarity matching with the pre-stored text feature library, and returns the Top-K text results with the highest relevance; similarly, when querying text, the system converts the input text into a feature vector and searches in the image feature library; the system of the present invention can not only realize cross-modal image-text mutual retrieval, but also support the use of fused features for single-modal image-image and text-text retrieval, showing strong versatility; through the three-stage method of distance correlation measurement, attention distillation network, and clustering optimization, high precision, high efficiency and multi-task compatibility of cross-modal retrieval are achieved, and it has broad application prospects.

[0167] Specifically, in this embodiment, the present invention overcomes the limitations of traditional Euclidean distance or cosine similarity by designing a distance correlation measurement module and calculating the inter-modal correlation by the difference between the joint distribution and the marginal distribution; and proposes a multi-level attention distillation network, which refines cross-modal features layer by layer by combining residual connections with knowledge distillation, and uses the cross-attention mechanism to achieve fine-grained feature alignment; further constructs an end-to-end joint optimization framework, while minimizing the intra-class distance correlation and maximizing the inter-class distance correlation, to improve the semantic consistency of cross-modal retrieval; the cross-modal retrieval method proposed in the present invention achieves a significant improvement in cross-modal retrieval performance while maintaining the integrity of single-modal retrieval capabilities, and realizes indiscriminate clustering of image and text modalities by constructing a unified feature space, effectively solving the problem of strong and weak modal imbalance caused by modal heterogeneity in traditional methods. At the same time, through the original feature alignment mechanism and lightweight network architecture, the system demonstrates significant technical advantages under standard test environments and can be widely used in scenarios requiring multimodal processing such as intelligent search and content recommendation.

[0168] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. All equivalent structural transformations made using the contents of the present description and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A cross-modal image and text retrieval method, characterized in that: The following steps are involved: S1. Constructing a set of image-text data pairs, wherein the image-text data pairs include data in an image modality and data in a corresponding text modality; and performing feature extraction on the data in the image modality and the data in the text modality respectively; S2. Extract hierarchical features of images and texts through a multi-level attention distillation network, and use cross-attention mechanism and knowledge distillation to perform fine-grained alignment between modalities to obtain fused features; S3. Use a dynamic batch clustering strategy to optimize feature alignment. Randomly select one category from each training batch as the positive category, and the remaining categories as negative categories. Calculate the intra-class distance of the positive category and the distance between positive and negative categories. Use the distance correlation coefficient to calculate feature similarity. Construct a joint loss function. By minimizing the loss function, minimize the intra-class distance correlation of similar samples and maximize the inter-class distance correlation of heterogeneous samples. S4. Map the fused features to the common feature space, calculate the similarity with the target modality features, and output the cross-modal and single-modal retrieval results based on the similarity, that is, output the cross-modal and single-modal retrieval results of images and texts.

2. A cross-modal image-text retrieval method according to claim 1, characterized in that: In step S1, the VGGNet or AlexNet model is used to extract image features of the image modality data.

3. A cross-modal image-text retrieval method according to any one of claims 1-2, characterized in that: In step S1, the BoW model is used to extract text features of text modal data.

4. A cross-modal image-text retrieval method according to any one of claims 1-2, characterized in that: The step S2 is specifically as follows: S21, performing multi-level processing on the input image features and text features respectively through a multi-level attention distillation network; the multi-level attention distillation network includes an image feature learning network and a text feature learning network; S22, generating a first image query vector, a first key vector, a first value vector, and a second text query vector, a second key vector, and a second value vector respectively according to the image features and text features of each level; S23. Generate an inter-modal alignment weight vector by calculating a similarity score between the second value vector and the second key vector, and perform a weighted summation of the weight vector and each row of the first value vector to obtain a new image feature; S24, performing linear transformation on the image features of each level to obtain a feature matrix, and performing linear transformation and normalization processing to generate weight coefficients, and obtaining the final image fusion feature through weighted summation; S25. Symmetrically process the text features and image features after multi-level processing by the multi-layer attention distillation network to obtain the final text fusion features.

5. A cross-modal image-text retrieval method according to claim 4, characterized in that: The image features are: in, is the image feature, i is the sample index, j is the level index, is the first value vector, is the first value vector The s-th row vector after reshaping, s is the number of rows, is the weight vector, is the second value vector With the first key vector The sth row after reshaping The similarity of is the second value vector, is the first key vector The mth row vector of .

6. A cross-modal image-text retrieval method according to claim 5, characterized in that: The final image fusion feature is: V i fuse =W v F i v Among them, V i fuse is the final image fusion feature, W v is the weight coefficient, F i v is the characteristic matrix; f fusel () is the image feature linear layer, These are the image features corresponding to levels 1, 2, and 3, respectively. is the parameter corresponding to the linear layer of image features.

7. A cross-modal image-text retrieval method according to any one of claims 1-2, characterized in that: In step S3, the intra-class distance of the positive class is calculated as follows: Calculate the mean vector of the positive class; specifically: calculate the mean vector of the image features respectively and the text feature mean vector And the image feature mean vector It is a positive category center; Calculate the mean vector of image features The dual-centered distance matrix Double-neutralized distance matrix of image features And the double-neutralized distance matrix of text features Calculate the intra-class distance correlation to obtain the intra-class distance of the positive class. Use the intra-class distance to quantify the average correlation between the image and text features within the positive class and the positive class center. The intra-class distance is: in, is the intra-class distance, m1 is the number of positive image-text pairs in a batch, and dCorr() represents the linear and nonlinear dependencies between features. is the mean vector of image features, is the image feature of the positive class, is the text feature of the positive class, k is the category index, i is the sample index, and j is the level index.

8. A cross-modal image-text retrieval method according to any one of claims 1-2, characterized in that: The distance between positive and negative classes is: in, is the distance between positive and negative classes, m2 is the amount of negative class image-text pairs in a batch, dCorr() represents the linear and nonlinear dependency between features, is the mean vector of image features, is the image feature of the negative class, is the text feature of the negative class, M is a category set in a batch, k is the category index, i is the sample index, and j is the level index.

9. A cross-modal image-text retrieval method according to any one of claims 1-2, characterized in that: The step S3 constructs a joint loss function, which minimizes the intra-class distance correlation of samples of the same class and maximizes the inter-class distance correlation of samples of different classes by minimizing the loss function. Specifically, Construct the j-th layer clustering loss function about the image center And the text features are processed symmetrically, with the text feature mean vector is the positive class center, and the clustering loss function of the jth layer about the text center is obtained The clustering loss of the jth layer is L j ; Calculate the clustering loss of the fusion feature, and get the total clustering loss L according to the clustering loss of each layer and the clustering loss of the fusion feature cluster ; Based on the total clustering loss L cluster , construct a total loss optimization model, and by minimizing the loss function, minimize the intra-class distance correlation of similar samples and maximize the inter-class distance correlation of heterogeneous samples.

10. A system comprising the image-text cross-modal retrieval method according to any one of claims 1 to 9, characterized in that: include: The data input and feature extraction module, feature interaction enhancement module, semantic optimization module, and retrieval execution module are connected in sequence; The data input and feature extraction module is used to receive image data and text data, and extract image features and text features of image-text data pairs; The feature interaction enhancement module is used to extract hierarchical features of image features and text features and perform modality alignment; The semantic optimization module is used to optimize feature alignment through a dynamic batch clustering strategy and perform loss calculation. By minimizing the loss function, the intra-class distance correlation of similar samples is minimized and the inter-class distance correlation of heterogeneous samples is maximized. The retrieval execution module is used for cross-modal image and text retrieval and single-modal retrieval.

Images