Cross-modal image-text retrieval method based on correlation matching

Abstract

This invention relates to the field of image and text retrieval technology, and discloses a cross-modal image and text retrieval method based on relevance matching, comprising the following specific steps: S1: extracting features from training images and text; S2: obtaining image embedding space features and text embedding space features; S3: projecting the image embedding space features and text embedding space features onto a label space; calculating a total loss function in the label space, including discriminative loss, matching relationship loss, and relevance loss; S4: training a feature extractor and a shared classifier based on the total loss function; S5: acquiring the multimodal data to be retrieved, and using the trained feature extractor and shared classifier to project the multimodal data features onto a shared subspace for cross-modal image and text retrieval. This invention solves the problem of differences in multimodal features within subspaces in existing technologies, and has the advantages of reducing computational time complexity and improving retrieval efficiency.

Description

Technical Field

[0001] This invention relates to the field of image and text retrieval technology, and more specifically, to a cross-modal image and text retrieval method based on relevance matching. Background Technology

[0002] Cross-modal retrieval technology plays a crucial role in improving the accuracy and efficiency of image and text retrieval, audio and video retrieval in search engines, and multimedia information management. The aim of this technology is to enhance the accuracy and efficiency of cross-modal retrieval.

[0003] For cross-modal retrieval problems, the mainstream techniques currently available are based on subspace learning frameworks and their improvements. Subspace learning methods aim to project multimodal data features into a shared subspace, removing redundant information between modalities. Key representative methods include Adversarial Cross-Modal Retrieval (ACMR) and Deep Supervised Cross-Modal Retrieval (DSCMR).

[0004] The adversarial cross-modal retrieval method (Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic Hengtao Shen. "Adversarial cross-modal retrieval." Proceedings of the 25th ACM International Conference on Multimedia. 2017.) designs a combination of modality encoder and modality discriminator. Through adversarial training, a modality-invariant latent feature subspace is obtained. At the same time, triplet constraints are added to the projection function to minimize the distance between samples from different modalities with the same semantic label, while maximizing the distance between samples with different semantics, thus ensuring the discriminativeness of the projection subspace.

[0005] The Deep Supervised Cross-Modal Retrieval (DSCMR) method (Liangli Zhen, Peng Hu, Xu Wang, Dezhong Peng. "Deep supervised cross-modal retrieval." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019.) designs two modality feature extractors and a shared classifier. It projects data from different modalities into a shared subspace and performs supervised training in the label space using the shared classifier. A weight-sharing strategy is employed to eliminate multimodal feature differences in the shared subspace. DSCMR also incorporates a modality consistency loss to reduce modality discrepancies from a distance perspective in Euclidean space.

[0006] The above methods project data from different modalities into a shared subspace using a modality encoder, obtaining modality-consistent features through projection, thereby achieving cross-modal retrieval. However, they still have the following shortcomings: Firstly, these methods ignore the specific matching relationships of multimodal samples, making it difficult to learn higher-level feature discriminative abilities. Therefore, it is necessary to incorporate such prior information to assist learning during the training process. Secondly, the above methods construct a modality dissimilarity measure based on Euclidean distance, without considering the statistical properties of the feature data. As a result, the learned multimodal features will still exhibit dissimilarity in the subspace.

[0007] Existing technology includes a cross-modal hashing retrieval method based on supervised graph embedding, belonging to the field of multimedia retrieval technology. This method constructs image and text modal datasets, dividing them into training and test sets. Features are extracted from all image and text samples in both sets, and the extracted features are mapped to a nonlinear kernel space using a radial basis function (RBF) kernel to enhance their representational power. Pairwise similarity matrices are generated using class labels, which in turn generate Laplacian matrices. A semantic subspace is generated using the class labels. A mapping matrix is ​​learned for the image and text modalities using intra-modal similarity preservation based on graph embedding and inter-modal similarity preservation based on the semantic subspace. An orthogonal rotation matrix is ​​learned to minimize quantization error. An efficient iterative discrete optimization algorithm reduces the computational complexity of the training process.

[0008] However, existing technologies suffer from the problem of differences in multimodal features within subspaces. Therefore, how to invent a cross-modal image and text retrieval method that takes into account the statistical attributes of feature data is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0009] To address the issue of differences in multimodal features across subspaces in existing technologies, this invention provides a cross-modal image and text retrieval method based on relevance matching, which features reduced computational time complexity and improved retrieval efficiency.

[0010] To achieve the above-mentioned objectives of this invention, the technical solution adopted is as follows:

[0011] The cross-modal image-text retrieval method based on relevance matching includes the following specific steps:

[0012] S1: Acquire training images and text; extract features from the training images and text to obtain image features and text features;

[0013] S2: Construct two feature extractors and one shared classifier corresponding to image features and text features, respectively; input the image features and text features into their corresponding feature extractors to obtain image embedding space features and text embedding space features;

[0014] S3: Project the image embedding spatial features and text embedding spatial features to the label space using a shared classifier; calculate the total loss function in the label space, including discriminative loss, matching relationship loss, and relevance loss;

[0015] S4: Train the feature extractor and the shared classifier based on the total loss function. During training, update the network parameters of the feature extractor and the shared classifier through backpropagation.

[0016] S5: Obtain the multimodal data to be retrieved, and use the trained feature extractor and shared classifier to project the multimodal data features into the shared subspace to perform cross-modal image and text retrieval.

[0017] Preferably, in step S1, feature extraction is performed on the training image and text, specifically through the following steps:

[0018] S101: Input n training images into the VGGNet network to obtain the n*4096-dimensional image features u output by the FC7 layer. i Input n training texts into the Doc2Vec model to obtain n*300 dimensional text features v. i ;

[0019] S102: For u i and v i Normalization is performed.

[0020] Furthermore, in step S2, constructing two feature extractors corresponding to the image features and text features specifically involves:

[0021] Construct a two-layer fully connected network as an image feature extractor f u (·), whose activation function is ReLU;

[0022] Construct a two-layer fully connected network as a text feature extractor f v (·), whose activation function is ReLU.

[0023] Furthermore, the shared classifier P(·) is composed of a single-layer fully connected network.

[0024] Furthermore, in step S2, the image features and text features are respectively input into their corresponding feature extractors to obtain image embedding space features and text embedding space features, specifically:

[0025] U i =f u (u i ),V i =f v (v i ),

[0026] Among them, U i For image embedding spatial features, V i For text embedding spatial features, U i V i ∈R d d represents the dimension of the shared subspace.

[0027] Furthermore, in step S3, the total loss function, which includes discriminative loss, matching relationship loss, and relevance loss, is calculated in the label space, specifically as follows:

[0028] Calculate the discriminant loss:

[0029] Let the label information of the training samples be Y∈R c Y is a 0-1 one-hot vector, and U i V i The discriminative loss J is calculated by projecting the shared classifier P(·) onto the label space. d :

[0030]

[0031] Where U,V∈R n*d Let ||·|| be the feature matrix composed of all samples. F Denotes the Frobenius norm;

[0032] Calculate the matching relationship loss:

[0033] Let the prior distribution of the matching relationship between samples be:

[0034]

[0035] The matching probability between features, calculated using Euclidean distance, is defined as follows:

[0036]

[0037] Calculate U i V i The Kullback-Leibler divergence between the two distributions yields the matching loss in the first direction:

[0038]

[0039] Similarly, by changing the order of U and V, the matching loss in the second direction is calculated:

[0040]

[0041] The matching loss J is obtained by adding the matching losses in the first and second directions. m ;

[0042] J m =J v2u +J u2v ;

[0043] Calculate the correlation loss:

[0044] For U i V i Perform mean normalization on the features:

[0045]

[0046]

[0047] Calculate the approximate correlation loss for low-rank rows:

[0048]

[0049] in The covariance matrix represents the image features and text features, and Tr(·) represents the trace operation;

[0050] Calculate the total loss function:

[0051] J = J d +αJ m +βJ c ,

[0052] Where α and β are hyperparameters.

[0053] Furthermore, in step S4, the network parameters of the feature extractor and the shared classifier are updated via backpropagation during training. Specifically, the Adam optimizer is used, and the network parameters of the feature extractor and the shared classifier are updated via backpropagation using the PyTorch machine learning library.

[0054]

[0055]

[0056]

[0057] Where θ u ,θ v ,θ P Image feature extractor f u (·), text feature extractor f v (·), the network parameters of the shared classifier P(·), and λ is the learning rate.

[0058] Furthermore, in step S5, the trained feature extractor and shared classifier are used to project the multimodal data features into a shared subspace for cross-modal image-text retrieval. The specific steps are as follows:

[0059] S501: Extract features from multimodal data to obtain its image features u t Text features v t ;

[0060] S502: u t Input the trained image feature extractor f u (·), v t Input text feature extractor f v In (·), its image embedding spatial features U are obtained. t and text embedding spatial features V t :

[0061] U t =f u (u t ),V t =f v (v t ).

[0062] S503: For any feature sample or m represents the number of features. Cosine similarity is calculated, all cosine similarities are sorted, and samples with the highest similarity ranking are selected for matching, thereby performing cross-modal retrieval from image to text or from text to image.

[0063] Furthermore, in step S503, the cosine similarity is calculated as follows:

[0064]

[0065] in, Let be the cosine similarity.

[0066] Furthermore, in step S4, during training, hyperparameters α = 0.01, β = 0.3 are selected, and the Adam optimizer with learning rates λ = 0.0001, β1 = 0.9, and β2 = 0.999 is selected.

[0067] The beneficial effects of this invention are as follows:

[0068] This invention designs a cross-modal image-text retrieval method based on relevance matching, solving the semantic mapping problem of intra-modal data. By constructing two feature extractors corresponding to image features and one shared classifier corresponding to text features, this invention projects image embedding spatial features and text embedding spatial features onto a label space, and calculates a total loss function in the label space, including discriminative loss, matching relationship loss, and relevance loss. It implements a low-rank approximation relevance learning loss for modal discriminative learning, better preserving the semantic information and statistical properties of the projected features. Therefore, this invention solves the problem of discriminative features in subspaces in existing technologies, and has the advantages of reducing computational time complexity and improving retrieval efficiency. Attached Figure Description

[0069] Figure 1 This is a flowchart illustrating the cross-modal image and text retrieval method based on relevance matching according to the present invention.

[0070] Figure 2 This is a flowchart illustrating the specific implementation of the cross-modal image and text retrieval method based on relevance matching in Example 2 of the present invention.

[0071] Figure 3 This is the original image feature map in Example 2.

[0072] Figure 4 This is the original text feature map in Example 2.

[0073] Figure 5 These are the original image and text feature maps from Example 2.

[0074] Figure 6 This is the image embedding spatial feature map in Example 2.

[0075] Figure 7 This is the text embedding spatial feature map in Example 2.

[0076] Figure 8 This is the spatial feature map of image and text embedding in Example 2. Detailed Implementation

[0077] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0078] Example 1

[0079] like Figure 1 As shown, the cross-modal image and text retrieval method based on relevance matching includes the following specific steps:

[0080] S1: Acquire training images and text; extract features from the training images and text to obtain image features and text features;

[0081] S2: Construct two feature extractors and one shared classifier corresponding to image features and text features, respectively; input the image features and text features into their corresponding feature extractors to obtain image embedding space features and text embedding space features;

[0082] S3: Project the image embedding spatial features and text embedding spatial features to the label space using a shared classifier; calculate the total loss function in the label space, including discriminative loss, matching relationship loss, and relevance loss;

[0083] S4: Train the feature extractor and the shared classifier based on the total loss function. During training, update the network parameters of the feature extractor and the shared classifier through backpropagation.

[0084] S5: Obtain the multimodal data to be retrieved, and use the trained feature extractor and shared classifier to project the multimodal data features into the shared subspace to perform cross-modal image and text retrieval.

[0085] In this embodiment, unlike the discriminative learning method based on label information, the present invention incorporates prior information of matching relationship loss into the discriminative learning process, uses the distance between samples to estimate the sample distribution, and learns the sample matching relationship by minimizing the divergence between the prior distribution and the sample conditional distribution. This makes the distance between matching samples close and the distance between non-matching samples far, thereby enhancing the discriminativeness of subspace features and ensuring the accuracy of retrieval.

[0086] In this embodiment, unlike the traditional modality difference measurement learning method based on Euclidean distance, this invention proposes a low-rank approximation correlation loss term. The correlation loss is based on the modality difference measurement of correlation, which can reduce computational complexity and theoretically guarantee the geometric properties of the projection function. This ensures that the subspace spanned by the basis vectors of the original projection is the same, thereby preserving the geometric structure and statistical information of multimodal features and improving retrieval accuracy and efficiency.

[0087] Example 2

[0088] In this embodiment, the present invention uses three publicly available cross-modal retrieval databases to demonstrate the implementation methods and effects of the present invention. The three datasets are: the Wikipedia dataset from Wikipedia's "Featured Articles" (source: http: / / www.svcl.ucsd.edu / projects / crossmodal / ), the Pascal Sentences dataset from the 2008 PASCAL development kit (source: https: / / vision.cs.uiuc.edu / pascal-sentences / ), and the PKU XMediaNet dataset from the Multimedia Information Processing Laboratory of Peking University (source: http: / / www.icst.pku.edu.cn / mipl / XMedia / ).

[0089] More specifically, in one embodiment, step S1 involves feature extraction from the training image and text, specifically through the following steps:

[0090] S101: Input n training images into the VGGNet network to obtain the n*4096-dimensional image features u output by the FC7 layer. i Input n training texts into the Doc2Vec model to obtain n*300 dimensional text features v. i In this embodiment, image feature u i and text features v i Their individual and common distributions are as follows Figure 3 , 4 As shown in Figure 5.

[0091] S102: For u i and v i Normalization is performed:

[0092]

[0093]

[0094] Where ||·||2 represents the l2 norm.

[0095] In one specific embodiment, step S2 involves constructing two feature extractors corresponding to the image features and the text features, specifically as follows:

[0096] Construct a two-layer fully connected network as an image feature extractor f u (·), whose activation function is ReLU; in this embodiment, f u In (·), the number of neurons in the input layer is 4096, the number of neurons in the hidden layer is 2048, and the number of neurons in the output layer is 1024.

[0097] Construct a two-layer fully connected network as a text feature extractor f v (·), whose activation function is ReLU; in this embodiment, f v In the diagram (·), the number of neurons in the input layer is 300, the number of neurons in the hidden layer is 2048, and the number of neurons in the output layer is 1024.

[0098] In one specific embodiment, the shared classifier P(·) is composed of a single-layer fully connected network; in this embodiment, the number of input layer neurons in P(·) is 1024, and the number of output layer neurons is c, which is the number of categories.

[0099] In one specific embodiment, in step S2, the image features and text features are respectively input into their corresponding feature extractors to obtain image embedding spatial features and text embedding spatial features, specifically:

[0100] U i =f u (u i ),V i =f v (v i ),

[0101] Among them, U i For image embedding spatial features, V i For text embedding spatial features, U i V i ∈R d d represents the shared subspace dimension. In this embodiment, the individual and common distributions of image embedding spatial features and text embedding spatial features are as follows: Figure 6 , Figure 7 , Figure 8 .

[0102] like Figure 2 As shown, in a specific embodiment, step S3 involves calculating a total loss function in the label space, including discriminative loss, matching relationship loss, and relevance loss, specifically as follows:

[0103] Calculate the discriminant loss:

[0104] Let the label information of the training samples be Y∈R c Y is a 0-1 one-hot vector, and U i V i The discriminative loss J is calculated by projecting the shared classifier P(·) onto the label space. d :

[0105]

[0106] Where, U,V∈R n*dLet ||·|| be the feature matrix composed of all samples. F Denotes the Frobenius norm;

[0107] Calculate the matching relationship loss:

[0108] Let the prior distribution of the matching relationship between samples be:

[0109]

[0110] The matching probability between features, calculated using Euclidean distance, is defined as follows:

[0111]

[0112] In this embodiment, the matching probability means that if sample U i and sample V j If a match is found, the probability is 1; otherwise, the probability is 0.

[0113] Calculate U i V i The Kullback-Leibler divergence between the two distributions yields the matching loss in the first direction:

[0114]

[0115] In this embodiment, the matching relationship loss in the first direction represents the loss if sample U i and sample V j The smaller the distance between them, the closer the matching probability p is to 1, and vice versa.

[0116] Similarly, by changing the order of U and V, the matching loss in the second direction is calculated:

[0117]

[0118] The matching loss J is obtained by adding the matching losses in the first and second directions. m ;

[0119] J m =J v2u +J u2v ;

[0120] Calculate the correlation loss:

[0121] For U i V i Perform mean normalization on the features:

[0122]

[0123]

[0124] Calculate the approximate correlation loss for low-rank rows:

[0125]

[0126] in The covariance matrix represents the image features and text features, and Tr(·) represents the trace operation;

[0127] Calculate the total loss function:

[0128] J = J d +αJ m +βJ c ,

[0129] Where α and β are hyperparameters.

[0130] In one specific embodiment, in step S4, the network parameters of the feature extractor and the shared classifier are updated via backpropagation during training. Specifically, the Adam optimizer is used, and the network parameters of the feature extractor and the shared classifier are updated via backpropagation based on the PyTorch machine learning library.

[0131]

[0132]

[0133]

[0134] Where θ u ,θ v ,θ P Image feature extractor f u (·), text feature extractor f v (·), the network parameters of the shared classifier P(·), and λ is the learning rate.

[0135] In one specific embodiment, in step S5, the trained feature extractor and shared classifier are used to project multimodal data features into a shared subspace for cross-modal image-text retrieval. The specific steps are as follows:

[0136] S501: Extract features from multimodal data to obtain its image features u t Text features v t ;

[0137] S502: u t Input the trained image feature extractor f u (·), v t Input text feature extractor f v In (·), its image embedding spatial features U are obtained. tand text embedding spatial features V t :

[0138] U t =f u (u t ),V t =f v (v t ).

[0139] S503: For any feature sample or m represents the number of features. Cosine similarity is calculated, all cosine similarities are sorted, and samples with the highest similarity ranking are selected for matching, thereby performing cross-modal retrieval from image to text or from text to image.

[0140] In one specific embodiment, step S503, calculating the cosine similarity, specifically involves:

[0141]

[0142] in, Let be the cosine similarity.

[0143] In one specific embodiment, in step S4, during training, hyperparameters α = 0.01, β = 0.3, and an Adam optimizer with learning rates λ = 0.0001, β1 = 0.9, and β2 = 0.999 are selected.

[0144] Example 3

[0145] More specifically, in this embodiment, the factual results obtained by using the cross-modal image and text retrieval method based on relevance matching of the present invention relative to the ACMR method and the DSCMR method are shown in Table 1:

[0146] Table 1

[0147]

[0148] In the table, i2t represents image retrieval of text, t2i represents text retrieval of image, and avg represents the average of the two tasks. The larger the number in the table, the better the implementation effect of the corresponding method. As shown in Table 1, the retrieval accuracy of the present invention is the best on both the large dataset PKUXMediaNet and the small datasets Wikipedia and Pascal Sentences. This verifies the effectiveness of the matching relationship learning and relevance learning objective function proposed in the present invention, which can make the features in the embedded subspace more discriminative and reduce the modal differences of multimodal data.

[0149] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A cross-modal image and text retrieval method based on relevance matching, characterized in that: The specific steps include the following: S1: Acquire training images and text; extract features from the training images and text to obtain image features and text features; S2: Construct two feature extractors and one shared classifier corresponding to image features and text features, respectively; input the image features and text features into their corresponding feature extractors to obtain image embedding space features and text embedding space features; Specifically: in, For image embedding spatial features, For text embedding spatial features, , To share subspace dimensions; S3: Project the image embedding spatial features and text embedding spatial features onto the label space using a shared classifier; calculate the total loss function in the label space, including discriminative loss, matching relationship loss, and relevance loss; specifically: Calculate the discriminant loss: Let the label information of the training samples be... , Y As a 0-1 one-hot vector, Use a shared classifier Projected onto the label space, the discriminative loss is calculated. : in The feature matrix consists of all samples. Denotes the Frobenius norm; Calculate the matching relationship loss: Let the prior distribution of the matching relationship between samples be: The matching probability between features, calculated using Euclidean distance, is defined as follows: calculate The Kullback-Leibler divergence between the two distributions yields the matching loss in the first direction: Similarly, by changing the order of U and V, the matching loss in the second direction is calculated: The matching loss is obtained by adding the matching losses in the first and second directions. ; Calculate the correlation loss: right Perform mean normalization on the features: Calculate the approximate correlation loss for low-rank rows: in The covariance matrix representing image features and text features. Represents trace operation; Calculate the total loss function: in For hyperparameters; S4: Train the feature extractor and the shared classifier based on the total loss function. During training, update the network parameters of the feature extractor and the shared classifier through backpropagation. S5: Obtain the multimodal data to be retrieved, and use the trained feature extractor and shared classifier to project the multimodal data features into the shared subspace to perform cross-modal image and text retrieval.

2. The cross-modal image and text retrieval method based on relevance matching according to claim 1, characterized in that: In step S1, feature extraction is performed on the training images and text. The specific steps are as follows: S101: Input n training images into the VGGNet network to obtain the n*4096-dimensional image features output by the FC7 layer. Inputting n training texts into the Doc2Vec model yields n*300 dimensional text features. ; S102: Yes and Normalization is performed.

3. The cross-modal image and text retrieval method based on relevance matching according to claim 2, characterized in that: In step S2, two feature extractors corresponding to image features and text features are constructed, specifically by building a two-layer fully connected network as the image feature extractor. Its activation function is ReLU; Construct a two-layer fully connected network as a text feature extractor Its activation function is ReLU.

4. The cross-modal image and text retrieval method based on relevance matching according to claim 3, characterized in that: The shared classifier It consists of a single-layer fully connected network.

5. The cross-modal image and text retrieval method based on relevance matching according to claim 4, characterized in that: In step S4, the network parameters of the feature extractor and the shared classifier are updated via backpropagation during training. Specifically, the Adam optimizer is used, and the network parameters of the feature extractor and the shared classifier are updated via backpropagation based on the PyTorch machine learning library. in Image feature extractors Text feature extractor Shared classifier Network parameters, This is the learning rate.

6. The cross-modal image and text retrieval method based on relevance matching according to claim 5, characterized in that: In step S5, the trained feature extractor and shared classifier are used to project multimodal data features into a shared subspace for cross-modal image-text retrieval. The specific steps are as follows: S501: Perform feature extraction on multimodal data to obtain the image features to be retrieved. Text features to be retrieved ; S502: Will Input the trained image feature extractor ,Will Input text feature extractor In this process, the spatial embedding features of the image to be retrieved are obtained. and the text embedding spatial features to be retrieved : S503: For any feature sample or , m is the number of features to be retrieved. Cosine similarity is calculated, all cosine similarities are sorted, and samples with the highest similarity ranking are selected for matching, thereby performing cross-modal retrieval from image to text or from text to image.

7. The cross-modal image and text retrieval method based on relevance matching according to claim 6, characterized in that: In step S503, the cosine similarity is calculated, specifically as follows: in, Let be the cosine similarity.

8. The cross-modal image and text retrieval method based on relevance matching according to claim 7, characterized in that: In step S4, hyperparameters are selected during training. Select learning rate , , Adam optimizer.

Images