A cross-modal retrieval method based on deep recognition transfer

Abstract

The present invention discloses a cross-modal retrieval method based on deep recognition transfer, which belongs to the field of information retrieval technology. In the process of learning the common representation features of different modal data, the method enhances the semantic recognition of the common representation features of the text by minimizing the reconstruction loss between the decoding vector and the original features of the text, and at the same time transfers the high semantic recognition of the common representation features of the text to the common representation features of the picture through the modality invariant loss and sample correlation loss of the weight sharing layer, thereby learning the common representation features with high semantic recognition for each modal data, thereby improving the retrieval performance.

Description

Technical Field

[0001] The present invention relates to a cross-modal retrieval method based on deep recognition transfer, and belongs to the technical field of information retrieval. Background Art

[0002] Cross-modal retrieval aims to achieve information retrieval across different modalities, that is, it can return retrieval results of other modalities related to the query sample based on samples of any query modality, for example, outputting corresponding image retrieval results based on the query text. In recent years, deep learning has made great progress in the field of cross-modal retrieval, and a series of cross-modal retrieval methods based on deep learning have been proposed. These cross-modal deep methods mine the correlation information between different modal data through deep autoencoding, generative adversarial methods, etc., bridge the semantic gap within the multimodal data modality and the heterogeneous gap between modalities, and learn semantically consistent common representation features.

[0003] However, existing cross-modal deep methods ignore the differences in semantic recognition between data of different modalities, resulting in the need for further improvement in their retrieval accuracy. Summary of the invention

[0004] In order to solve the problem of low retrieval accuracy caused by insufficient consideration of the semantic recognition differences of multimodal data in existing cross-modal retrieval methods based on deep learning, the present invention provides a cross-modal retrieval method based on deep recognition transfer. The method comprises:

[0005] Step S1, constructing a deep recognition transfer network, wherein the deep recognition transfer network includes an original multimodal feature extraction module, a text recognition transfer module, and a text recognition enhancement module, and each module is connected in sequence;

[0006] Step S2, using a public data set to train the constructed deep recognition transfer network, wherein the public data set includes n picture-text pairs;

[0007] Step S3, using the trained deep recognition transfer network to obtain the optimal common representation features of the image or text to be retrieved, and using the cosine distance to measure the distance between the optimal common representation features of the image or text to be retrieved and the common representation features of each image or text in the database, to obtain a similarity matrix, and output the corresponding cross-modal retrieval results according to the similarity matrix.

[0008] Optionally, when training the constructed deep recognition transfer network, the method first extracts the original features of the image and the original features of the text in the public dataset, and respectively learns the corresponding image public representation features U = [u1, u2, ... u n ] and the common representation feature V = [v1, v2, …v n]; then set the semantic constraint function J3 and the reconstruction loss function J4 to enhance the semantic recognition of the text common representation feature V; set the modality invariant loss function J1 and the sample relevance loss function J2 to transfer the enhanced semantic high recognition of the text common representation feature to the picture common representation feature, enhance the semantic recognition of the picture common representation feature, minimize the total objective function J during the training process and use the Adam model to update the parameters, and continuously iterate to obtain a trained deep recognition migration network, wherein the total objective function J is composed of the modality invariant loss function J1, the sample relevance loss function J2, the semantic constraint function J3 and the reconstruction loss function J4;

[0009] where u i The common representation feature vector of image i, v j The common representation feature vector representing text j, i = 1, 2, ..., n; j = 1, 2, ..., n.

[0010] Optionally, the original multimodal feature extraction module is composed of two symmetrical feature extraction parts, which are used to extract the original features of the picture and the text respectively, and obtain the original feature matrix H of the picture α And the original feature matrix H β ;

[0011] The text recognition transfer module is composed of a picture common representation feature learning module and a text common representation feature learning module; both are composed of two fully connected layers with ReLU activation function, and the last fully connected layers of the two share weights, which is called a shared weight layer;

[0012] The text recognition enhancement module is composed of a linear classifier of the common text representation features and a decoding module of the common text representation features; wherein the linear classifier of the common text representation features is a layer of full connection, and the decoding module of the common text representation features includes two layers of full connection with the activation function being ReLU.

[0013] Optionally, the method obtains the original feature matrix H of the picture α And the original feature matrix H β After that, it also includes:

[0014] Define the modality invariant loss function J1 and the sample correlation loss function J2 in the shared weight layer;

[0015]

[0016] Among them, ||·|| F represents the F-norm;

[0017]

[0018] in, represents the cosine similarity between the common representation features of image i and the common representation features of text j, u i The common representation feature vector of image i, v j The common representation feature vector representing text j; represents the cosine similarity between the common representation features of image i and the common representation features of image j, represents the cosine similarity between the common representation features of text i and the common representation features of text j, 1{·} is the indicator function;

[0019] The original feature matrix H of the picture α And the original feature matrix H β The text recognition transfer module is sent to obtain the corresponding image common representation features U and text common representation features V.

[0020] Optionally, the method further includes:

[0021] The linear classifier is used to predict the semantic category of the common representation features of the text in the common subspace, and a semantic constraint function J3 is added:

[0022]

[0023] Among them, P T V is the predicted category vector generated by the text modality sample, and Y is the true category label;

[0024] The decoding module of the text common representation feature uses an incomplete autoencoder to decode each text common representation feature v in the text common representation feature V. i Decode and get the corresponding text decoding features Then we get the text decoding feature matrix

[0025] Define the reconstruction loss function J4, and use the reconstruction loss function J4 to minimize the distance between the text decoding feature vector and the text original feature vector:

[0026]

[0027] The overall objective function of the cross-modal retrieval method for deep recognition transfer is set as:

[0028] J=λ1J1+λ2J2+λ3J3+λ4J4

[0029] Among them, λ1, λ2, λ3, and λ4 are balanced hyperparameters, which respectively represent the importance of modality invariance loss, sample correlation loss, text semantic loss, and text decoding feature reconstruction loss.

[0030] Minimize the total objective function J of the network and use the Adam model to update the parameters, and iterate continuously to obtain the trained deep recognition transfer network; the update formula of the linear classifier is P = (VV T ) -1 V T Y.

[0031] Optionally, the public data sets used in step S2 of the method include a Wikipedia data set and a PascalSentence data set.

[0032] Optionally, the balancing hyperparameters λ1, λ2, λ3, and λ4 are set to λ1=1, λ2=1, λ3=25, and λ4=1.

[0033] Optionally, the numbers of hidden units of the fully connected layers of the two activation functions of the image common representation feature learning module and the text common representation feature learning module are 2048 and 1024 respectively.

[0034] Optionally, the method further includes:

[0035] Get the label matrix Y corresponding to the Wikipedia dataset = {y1,y2,…y n}, the label vector y corresponding to each image-text pair in the Wikipedia dataset i =[y 1i ,y 2i ,…,y ci ]∈R 10 ;

[0036] Get the label matrix Y corresponding to the Pascal Sentence dataset = {y1,y2,…y n}, the label vector y corresponding to each image-text pair in the Pascal Sentence dataset i =[y 1i ,y 2i ,…,y ci ]∈R 20 ;

[0037] Where c represents the sample category.

[0038] Optionally, in step S2, when the deep recognition transfer network constructed by training a public data set is used, the learning rate τ is set to 0.0001 and the maximum number of iterations N is set to 1000.

[0039] The beneficial effects of the present invention are:

[0040] By proposing a new cross-modal retrieval method, the semantic recognition of text common representation features is enhanced through the semantic constraints and reconstruction constraints of the text recognition enhancement subnet, and the high semantic recognition of text common representation features is transferred to image common representation features through the modal invariance constraints and sample correlation constraints between image and text common representation features of the text recognition transfer subnet, thereby enhancing the semantic recognition of image common representation features, thereby improving cross-modal retrieval performance and better completing cross-modal retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 It is a schematic diagram of a deep recognition transfer network framework established in a cross-modal retrieval method based on deep recognition transfer provided in one embodiment of the present application.

[0043] Figure 2A It is a confusion matrix diagram of the image features of the Wikipedia dataset;

[0044] Figure 2B It is a confusion matrix diagram of text features of the Wikipedia dataset. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0046] Embodiment 1:

[0047] This embodiment provides a cross-modal retrieval method based on deep recognition transfer, the method comprising:

[0048] Step S1, constructing a deep recognition transfer network, wherein the deep recognition transfer network includes an original multimodal feature extraction module, a text recognition transfer module, and a text recognition enhancement module, and each module is connected in sequence;

[0049] Step S2, using a public data set to train the constructed deep recognition transfer network, wherein the public data set includes n picture-text pairs;

[0050] Step S3, using the trained deep recognition transfer network to obtain the optimal common representation features of the image or text to be retrieved, and using the cosine distance to measure the distance between the optimal common representation features of the image or text to be retrieved and the common representation features of each image or text in the database, to obtain a similarity matrix, and output the corresponding cross-modal retrieval results according to the similarity matrix.

[0051] Step S2: When training the constructed deep recognition transfer network, first extract the original features of the image and the original features of the text in the public dataset, and learn the corresponding image public representation features U = [u1, u2, ... u n ] and the common representation feature V = [v1, v2, …v n ]; then set the semantic constraint function J3 and the reconstruction loss function J4 to enhance the semantic recognition of the text common representation feature V; set the modality invariant loss function J1 and the sample relevance loss function J2 to transfer the enhanced semantic high recognition of the text common representation feature to the picture common representation feature, enhance the semantic recognition of the picture common representation feature, minimize the total objective function J during the training process and use the Adam model to update the parameters, and continuously iterate to obtain a trained deep recognition migration network, wherein the total objective function J is composed of the modality invariant loss function J1, the sample relevance loss function J2, the semantic constraint function J3 and the reconstruction loss function J4;

[0052] where u i The common representation feature vector of image i, v j The common representation feature vector representing text j, i = 1, 2, ..., n; j = 1, 2, ..., n.

[0053] Embodiment 2:

[0054] This embodiment provides a cross-modal retrieval method based on deep recognition transfer. This embodiment uses two internationally open and widely used cross-modal datasets, the Wikipedia dataset and the Pascal Sentence dataset, to train and test the cross-modal retrieval network based on deep recognition transfer constructed in the proposed cross-modal retrieval method based on deep recognition transfer.

[0055] The Wikipedia dataset contains 10 semantic categories and a total of 2,866 image-text pairs. In the Wikipedia dataset, the text descriptions paired with images consist of varying numbers of paragraphs. The Pascal Sentence dataset contains 20 semantic categories and a total of 1,000 image-text pairs. In the Pascal Sentence dataset, the text descriptions paired with images consist of five manually annotated sentences.

[0056] For the Wikipedia dataset, please refer to “A new approach to cross-modalmultimedia retrieval” by Rasiwasia N et al., which was published in “ACM International Conference on Multimedia” in 2010, pages 251-260.

[0057] For the Pascal Sentence dataset, please refer to "Collecting image annotations using Amazon's Mechanical Turk" by Rashtchian C et al., which was published in "Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk. Stroudsburg" in 2010, pages 139-147.

[0058] Please refer to Figure 2A and Figure 2B , extract the corresponding original underlying deep features for the images and text data in the Wikipedia dataset, which are 4096-dimensional image convolution features and 300-dimensional text convolution features, perform single-modal classification in each original space and draw the confusion matrix of the image modality and text modality. Figure 2A and Figure 2B It can be seen that the semantic recognition of the original text features is significantly stronger than that of the original image features. Therefore, the cross-modal retrieval method based on deep recognition transfer proposed in this application retains the semantic recognition of the text as much as possible when learning the common representation features of images and texts, and brings the recognition of the common subspace closer to the text space, which can improve the semantic recognition of the common representation features of different modalities, thereby improving the cross-modal retrieval performance.

[0059] The cross-modal retrieval method for deep recognition transfer described in the present invention comprises the following steps:

[0060] Step (1) Build a deep recognition transfer network:

[0061] Step 1.1: Please refer to Figure 1 The deep recognition transfer network constructed in the method described in the present invention includes three components, namely, the original multimodal feature extraction module, the text recognition transfer module and the text recognition enhancement module, and each module is connected in sequence.

[0062] Step 1.2: Reference Figure 1The original multimodal feature extraction module consists of two symmetrical feature extraction parts, which are used to extract the original features of images and texts respectively. The original feature extraction part of images is consistent with the VGG19 network pre-trained on ImageNet; the original feature extraction part of text is consistent with Text CNN.

[0063] The extracted original feature matrix of the image is recorded as Each element represents the original feature vector of an image in the dataset; the extracted text original feature matrix is ​​recorded as Each element represents the original feature vector of a text in the data set. In this application, superscripts α and β are used to distinguish images and texts.

[0064] For the Text CNN network, please refer to Kim Y et al.’s “Convolutional neural networks for sentence classification”, which was published in the Conference on Empirical Methods in Natural Language Processing in 2014, pages 1746-1751.

[0065] Step 1.3: Reference Figure 1 The text recognition transfer module consists of a picture common representation feature learning module and a text common representation feature learning module, which are used for; the picture common representation feature learning module and the text common representation feature learning module are both composed of two fully connected layers with ReLU activation function, and the number of hidden units are 2048 and 1024 respectively, and the last layer of the two modules is fully connected with shared weights (hereinafter referred to as shared weight layer).

[0066] The modality invariant loss function J1 and the sample correlation loss function J2 are defined in the shared weight layer. The modality invariant loss function J1 is used to minimize the distance between the common representation features of the paired images and the common representation features of the text to eliminate the modality differences between the common representation feature vectors of samples of different modalities, and is defined as:

[0067]

[0068] Among them, ||·|| F represents the F-norm, U is the common representation feature of the image, V is the common representation feature of the text, and n is the number of samples.

[0069] The sample correlation loss function J2 is used for the correlation between multimodal data modes and the local consistency within the mode, and is defined as:

[0070]

[0071] in, represents the cosine similarity between the common representation features of image i and the common representation features of text j, u i The common representation feature vector of image i, v j The common representation feature vector representing text j; represents the cosine similarity between the common representation features of image i and the common representation features of image j, represents the cosine similarity between the common representation features of text i and the common representation features of text j, 1{·} is an indicator function, which means that when two input samples are of the same modality and the same category, the value of the indicator function is 1, otherwise it is 0.

[0072] The first term of formula (2) can be used to measure the distance between the common representation features of the image and the common representation features of the text. The second and third terms are used to measure the similarity within the image modality sample and the text modality sample, respectively.

[0073] Step 1.4: Reference Figure 1 The text recognition enhancement module consists of a linear classifier of the common representation features of the text and a decoding module of the common representation features of the text, wherein the linear classifier of the common representation features of the text is a fully connected layer with the number of units consistent with the number of categories in the data set; the decoding module of the common representation features of the text contains two layers of fully connected layers with the activation function of ReLU, the number of units in the first layer is 10, and the number of units in the second layer is 300, which is consistent with the dimension of the original feature vector of the text. In the semantic loss part, this application only retains the text semantic loss, removes the semantic loss of the image, uses a linear classifier to predict the semantic category of the common representation features of the text in the common subspace, and adds a semantic constraint function J3, so that the predicted category vector P generated by the text modality sample T V should be as consistent as possible with the true category label Y, and the semantic constraint function J3 is defined as:

[0074]

[0075] The decoding module of the common representation feature of the text uses an incomplete autoencoder to decode the common representation feature v i Decode and get text decoding features

[0076] To make the text decode feature matrix Keep it consistent with the original feature matrix of the text as much as possible, and use the reconstruction loss to minimize the distance between the text decoding vector and the original feature vector of the text. The reconstruction loss function is defined as:

[0077]

[0078] Step 1.5 sets the overall objective function of the cross-modal retrieval method for deep recognition transfer as:

[0079] J=λ1J1+λ2J2+λ3J3+λ4J4

[0080] Among them, λ1, λ2, λ3, and λ4 are balanced hyperparameters, which respectively represent the importance of modality invariance loss, sample correlation loss, text semantic loss, and text decoding feature reconstruction loss.

[0081] Step (2) Dataset preprocessing:

[0082] Step 2.1: Randomly divide the Wikipedia dataset into training set, test set and validation set. The training set contains 2173 image-text pairs, the test set contains 462 image-text pairs, and the rest are validation sets.

[0083] The Pascal Sentence dataset is also randomly divided into two parts, where the training set contains 800 image-text pairs, the test set contains 100 image-text pairs, and the rest are validation sets.

[0084] Step 2.2: Get the label matrix Y={y1,y2,…y n}, the label vector y corresponding to each image-text pair in the Wikipedia dataset i =[y 1i ,y 2i ,…,y ci ]∈R 10 .

[0085] Get the label matrix Y corresponding to the Pascal Sentence dataset = {y1,y2,…y n}, the label vector y corresponding to each image-text pair in the Pascal Sentence dataset i =[y 1i ,y 2i ,…,y ci ]∈R 20 .

[0086] In this application, it is assumed that the sample categories in the Wikipedia dataset and the Pascal Sentence dataset are both c.

[0087] Step (3) Network training:

[0088] Step 3.1: Set the learning rate τ = 0.0001, the maximum number of iterations N = 1000, and the balance parameters of each loss λ1 = 1, λ2 = 1, λ3 = 25, and λ4 = 1.

[0089] Randomly initialize weights, including the trainable parameters γ of the image mapping function α , the trainable parameter γ of the text mapping function β And the linear classifier parameters P. The following steps are repeated for each iteration until 1000 iterations are completed.

[0090] Step 3.2: The image samples and text samples in the training set obtained in step 2.1 are sent into the deep recognition transfer network constructed in step (1) in two ways.

[0091] Step 3.3: For each image, use the VGG19 network pre-trained on ImageNet to extract its features and output the 4096-dimensional image features of the fc7 layer. Perform the same operation on the entire training set of images to obtain the original feature matrix H of the image. α .

[0092] For each text, we first use the Word2Vec model pre-trained on Google News to obtain the k-dimensional embedding vector representation of each word in the text. Then each text can be represented as an m×k-dimensional feature matrix, where m is the number of words in each text. Then, we feed the obtained text word vector into the Text CNN network and output the 300-dimensional text features of the pooling layer. We perform the same operation on the entire training set to obtain the original text feature matrix H. β .

[0093] Step 3.4: The original feature matrix H obtained in step 3.3 α and H β The text recognition transfer module is sent to learn the common representation features of multimodal data, that is, the extracted image features H α The image common representation feature learning module is used to obtain the image common representation feature U, and the extracted text feature H β Send it into the text common representation feature learning module to obtain the text common representation feature V.

[0094] Step 3.5: The text common representation feature V obtained in step 3.4 and the true label matrix Y obtained in step 2.2 are fed into the linear classifier of the text common representation feature in the text recognition enhancement module, and the text prediction category vector P is output. T V.

[0095] Step 3.6: Combine the common text representation feature V obtained in step 3.4 and the original text feature vector H obtained in step 3.3 β The text decoding module is sent to the text recognition enhancement subnet, and the text decoding vector D is generated using the incomplete autoencoder β .

[0096] Step 3.7: Minimize the total network loss J and use the Adam model to update the parameters, where the update formula of the linear classifier is P = (VV T ) -1 V T Y, trainable parameter of the image-text mapping function

[0097] For an introduction to the Adam model, please refer to: Kingma DP, Ba J. Adam: a method for stochastic optimization [EB / OL]. https: / / arxiv.org / abs / 1412.6980, 2014-12-22.

[0098] Step (4) Cross-modal retrieval accuracy test:

[0099] Step 4.1: Perform two cross-modal retrieval tasks, image retrieval text (I2T) and text retrieval image (T2I), on the two cross-modal datasets Wikipedia and Pascal Sentence, and feed the images and texts in the test set into the trained optimal model.

[0100] Step 4.2: Use the trained deep recognition transfer network to learn the optimal common representation features for the test set samples, and use the cosine distance to measure the distance between the common representation features of the image and text to obtain the similarity matrix.

[0101] Output the corresponding cross-modal retrieval results according to the similarity matrix.

[0102] To further evaluate the cross-modal retrieval performance of this application, continue with step 4.3;

[0103] Step 4.3: Calculate the AP value of each sample based on the similarity matrix obtained in step 4.2, and average the AP values ​​of all samples in the test set to obtain the final mAP value, which can evaluate the performance of the cross-modal retrieval method.

[0104] Step 4.4: In order to verify the detection performance of the method provided in this application, a comparative experiment was conducted on the two benchmark datasets mentioned above, including two traditional cross-modal retrieval methods, namely CCA and JRL, and six depth-based cross-modal retrieval algorithms, namely CMDN, CCL, DCCA, ACMR, DSCMR and MMCA-CMR.

[0105] Experimental results:

[0106] Table 1 shows the comparison results of mAP values ​​of cross-modal retrieval on the Wikipedia dataset.

[0107] Table 2 shows the comparison results of mAP values ​​for cross-modal retrieval on the Pascal Sentence dataset.

[0108] From the results in Tables 1 and 2, we can see that in the image retrieval text I2T task, the average mAP of this method on the Wikipedia dataset and the Pascal Sentence dataset are 0.574 and 0.760 respectively, and in the text retrieval image T2I task, the average mAP values ​​of this method on the Wikipedia dataset and the Pascal Sentence dataset are 0.506 and 0.772 respectively. The performance of each retrieval task has been improved to a certain extent, and both datasets have exceeded the 8 comparison methods listed above. In addition, DSCMR has the best performance among the 8 comparison methods. Compared with DSCMR, the average performance of the method provided in this application has increased by 0.7% and 3.0% on the two datasets respectively.

[0109] Table 1 Results on the Wikipedia dataset

[0110]

[0111] Table 2 Results of Pascal Sentence dataset

[0112]

[0113] The eight existing advanced cross-modal retrieval methods listed above are introduced as follows:

[0114] CCA can refer to Hotelling H's "Relations between two sets of variates", which was published in "Biometrika" on pages 321-377 in 1936. This method maximizes the correlation between paired data of different modes by means of typical variables to learn a set of mapping matrices.

[0115] JRL can refer to "Leaming crossmedia joint representation with sparse and semisupervised regularization" by Zhai XH et al., which was published in "IEEE Transactions on Circuits and Systems for Video Technology" in 2014, pages 965-978. This method builds a graph model for multimodal data to jointly mine semantic information and related information.

[0116] For CMDN, please refer to “Cross-media shared representation by hierarchical learning with multiple deep networks” by Peng YX et al., which was published in “International Joint Conference on Artificial Intelligence” in 2016, pages 3846-3853. This method fully exploits the information within and between modalities through joint learning of hierarchical networks.

[0117] CCL can refer to “CCL: Cross-modal correlation learning with multigrained fusion by hierarchical network” by Peng YX et al., which was published in IEEE Transactions on Multimedia in 2017, pages 405-420. This method combines CCA with a deep network to learn the nonlinear transformation relationship between two modal data.

[0118] DCCA can refer to "Deep canonical correlation analysis" by Andrew G et al., which was published in "International Conference on Machine Learning" in 2013, pages 1247-1255. This method combines CCA with a deep network to learn the nonlinear transformation relationship between two modal data.

[0119] ACMR can refer to "Adversarial cross-modal retrieval" by Wang BK et al., which was published in "ACM International Conference on Multimedia" in 2017, pages 154-162. This method learns the common representation features of multimodal data through a feature generator, and uses a modality discriminator to distinguish the original modality of the generated features to learn the modality-invariant representation of multimodal data.

[0120] DSCMR can refer to "Deep supervised cross-modal retrieval" by Zhen LL et al., which was published in "IEEE / CVF Conference on Computer Vision and Pattern Recognition" in 2019, pages 10394-10403. This method improves cross-modal retrieval performance by minimizing the discriminative loss of the common subspace and label space and the modality invariance loss between different modal data.

[0121] MMCA-CMR can refer to "Deep supervised multimodal semantic autoencoder for cross-modal retrieval" by Tian Y et al., which was published in Volume 31 of "Computer Animation and Virtual Worlds" in 2020. This method uses a multimodal semantic autoencoder to learn content-aware feature vectors to fully mine the relevant information between different modal data.

[0122] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0123] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A cross-modal retrieval method based on deep recognition transfer, characterized in that: The method comprises: Step S1, constructing a deep recognition transfer network, wherein the deep recognition transfer network includes an original multimodal feature extraction module, a text recognition transfer module, and a text recognition enhancement module, and each module is connected in sequence; Step S2, using a public data set to train the constructed deep recognition transfer network, wherein the public data set includes n picture-text pairs; Step S3, using the trained deep recognition transfer network to obtain the optimal common representation features of the image or text to be retrieved, and using the cosine distance to measure the distance between the optimal common representation features of the image or text to be retrieved and the common representation features of each image or text in the database to obtain a similarity matrix, and outputting the corresponding cross-modal retrieval results according to the similarity matrix; When training the constructed deep recognition transfer network, the method first extracts the original features of the image and the original features of the text in the public dataset, and learns the corresponding image public representation features U = [u1, u2, ... u n ] and the common representation feature V = [v1, v2, …v n ]; then set the semantic constraint function J3 and the reconstruction loss function J4 to enhance the semantic recognition of the text common representation feature V; set the modality invariant loss function J1 and the sample relevance loss function J2 to transfer the enhanced semantic high recognition of the text common representation feature to the picture common representation feature, enhance the semantic recognition of the picture common representation feature, minimize the total objective function J during the training process and use the Adam model to update the parameters, and continuously iterate to obtain a trained deep recognition migration network, wherein the total objective function J is composed of the modality invariant loss function J1, the sample relevance loss function J2, the semantic constraint function J3 and the reconstruction loss function J4; where u i The common representation feature vector of image i, v j The common representation feature vector representing text j, i = 1, 2, ..., n; j = 1, 2, ..., n; The original multimodal feature extraction module consists of two symmetrical feature extraction parts, which are used to extract the original features of the picture and the text respectively, and obtain the original feature matrix H of the picture α And the original feature matrix H β ; The text recognition transfer module is composed of a picture common representation feature learning module and a text common representation feature learning module; both are composed of two fully connected layers with ReLU activation function, and the last fully connected layers of the two share weights, which is called a shared weight layer; The text recognition enhancement module is composed of a linear classifier of the common representation features of the text and a decoding module of the common representation features of the text; wherein the linear classifier of the common representation features of the text is a layer of full connection, and the decoding module of the common representation features of the text includes two layers of full connection with the activation function being ReLU; The method obtains the original feature matrix H of the picture α And the original feature matrix H β After that, it also includes: Define the modality invariant loss function J1 and the sample correlation loss function J2 in the shared weight layer; Among them, ||·|| F represents the F-norm; in, represents the cosine similarity between the common representation features of image i and the common representation features of text j, u i The common representation feature vector of image i, v j The common representation feature vector representing text j; represents the cosine similarity between the common representation features of image i and the common representation features of image j, represents the cosine similarity between the common representation features of text i and the common representation features of text j, 1{·} is the indicator function; The original feature matrix H of the picture α And the original feature matrix H β Send it to the text recognition transfer module to obtain the corresponding image common representation feature U and text common representation feature V; The method further comprises: The linear classifier is used to predict the semantic category of the common representation features of the text in the common subspace, and a semantic constraint function J3 is added: Among them, P T V is the predicted category vector generated by the text modality sample, and Y is the true category label; The decoding module of the text common representation feature uses an incomplete autoencoder to decode each text common representation feature v in the text common representation feature V. i Decode and get the corresponding text decoding features Then we get the text decoding feature matrix Define the reconstruction loss function J4, and use the reconstruction loss function J4 to minimize the distance between the text decoding feature vector and the text original feature vector: The overall objective function of the cross-modal retrieval method for deep recognition transfer is set as: J=λ1J1+λ2J2+λ3J3+λ4J4 Among them, λ1, λ2, λ3, and λ4 are balanced hyperparameters, which respectively represent the importance of modality invariance loss, sample correlation loss, text semantic loss, and text decoding feature reconstruction loss; Minimize the total objective function J of the network and use the Adam model to update the parameters, and iterate continuously to obtain the trained deep recognition transfer network; the update formula of the linear classifier is P = (VV T ) -1 V T Y.

2. The method according to claim 1, characterized in that The public data sets used in step S2 of the method include the Wikipedia data set and the Pascal Sentence data set.

3. The method according to claim 2, characterized in that The balancing hyperparameters λ1, λ2, λ3, and λ4 are set to λ1=1, λ2=1, λ3=25, and λ4=1.

4. The method according to claim 3, characterized in that: The number of hidden units of the fully connected layers of the two activation functions of the image common representation feature learning module and the text common representation feature learning module is 2048 and 1024 respectively.

5. The method according to claim 4, characterized in that The method further comprises: Get the label matrix Y corresponding to the Wikipedia dataset = {y1,y2,…y n }, the label vector corresponding to each image-text pair in the Wikipedia dataset Get the label matrix Y corresponding to the Pascal Sentence dataset = {y1,y2,…y n }, the label vector corresponding to each image-text pair in the Pascal Sentence dataset Where c represents the sample category.

6. The method according to claim 5, characterized in that In step S2, when the deep recognition transfer network constructed by using the public data set for training is constructed, the learning rate τ is set to 0.0001 and the maximum number of iterations N is set to 1000.

Images