Robust hashing-based unpaired zero-shot cross-modal retrieval method for image-text

Abstract

The application discloses a robust hashing unpaired zero sample graph-text cross-modal retrieval method. The method can effectively solve the retrieval scene containing noise labels and unpaired simultaneously. The method designs a zero sample graph-text cross-modal hashing retrieval framework to learn different hash codes for image and text two modes, so that it is suitable for unpaired retrieval scene. In addition, the method respectively applies sparse constraint and low rank constraint on noise matrix and recovered label matrix, so as to effectively reduce the negative influence of noise labels. Then, epsilon drag is introduced in label semantic embedding learning, so as to learn more discriminative hash codes. In order to enhance the similarity semantic information in the hash code, the intra-modal similarity and inter-modal similarity are considered. The application improves the retrieval precision of unpaired zero sample graph-text cross-modal retrieval with noise labels.

Description

TECHNICAL FIELD

[0001] The application relates to a robust hashing unpaired zero sample graph-text cross-modal retrieval method and belongs to the zero sample graph-text cross-modal hashing retrieval field. BACKGROUND

[0002] In recent years, with the continuous development of network and technology, a large number of multimedia data including images, texts, audios and videos emerge on major social media. Although these multimedia data belong to different modalities, there are data expressing the same meaning among them. How to search for data expressing the same meaning from such multi-modal data is the current demand of information mining. In addition, with the continuous emergence of new concepts, we cannot train all classes. Therefore, under the condition that the model does not need to be retrained by new concept samples, achieving high-precision retrieval of new concepts (unseen classes) is the goal we want to achieve. However, most of the existing graph-text cross-modal hashing retrieval methods are to solve the standard graph-text cross-modal retrieval problem, that is, only applicable to the retrieval scene where the number of classes of the training set and the query set completely coincides. For the current emerging new concept graph-text cross-modal dataset, the original standard graph-text cross-modal retrieval method cannot well adapt to such a retrieval scene.

[0003] In order to achieve better retrieval performance in the retrieval scene containing new concepts, zero-shot hash retrieval methods have received more and more attention. Zero-shot hash retrieval methods attempt to use class attribute information to connect visible classes and invisible classes, thereby achieving the transfer of knowledge from visible classes to invisible classes. However, most of the existing zero-shot hash retrieval methods are proposed for single-modal retrieval tasks, and these methods cannot be directly applied to the image-text cross-modal zero-shot retrieval task. Methods proposed for the image-text cross-modal zero-shot retrieval task are constantly emerging. Some researchers use features and class attributes to learn hash codes simultaneously, and embed the local structure information of the modal in the hash code. Other researchers learn three different hash codes for images, texts and class attributes, and use the hash code corresponding to the class attribute to connect the image and text two modalities. This method not only narrows the gap between modalities, but also realizes the transfer of attribute information from visible classes to invisible classes. Although the existing image-text cross-modal zero-shot hash method has been applied to the zero-shot retrieval task, there are still some challenges that need to be further solved: 1) Most of the existing methods assume that the labels of the training data are complete and accurate. However, the labels required in training are marked by artificial, so there will be often omissions and incorrect class assignments in the marking process. 2) Secondly, most zero-shot image-text cross-modal hash retrieval methods assume that the data of all modalities are completely paired. However, in practical applications, the data of different modalities often have the problems of loss and alignment error, so it is unrealistic to assume that the data of different modalities are completely paired. 3) In addition, the existing zero-shot image-text cross-modal hash retrieval method does not fully utilize the discriminative ability of label information in the learning of label-guided hash codes, thereby affecting the retrieval performance. SUMMARY

[0004] In view of the above problems, which result in poor retrieval performance, the present application provides a robust hash unpaired zero-shot image-text cross-modal retrieval method, which improves the retrieval precision of unpaired zero-shot image-text cross-modal retrieval with noisy labels.

[0005] In order to achieve the object of the present application, the technical scheme of the robust hashing unpaired zero-shot cross-modal retrieval method for image and text is as follows: a zero-shot cross-modal hashing retrieval framework is designed to learn different hash codes for different modalities of images and texts, so that it is suitable for unpaired retrieval scenarios. In addition, sparse constraints and low-rank constraints are imposed on the noise matrix and the recovered label matrix, respectively, to effectively reduce the negative impact of noisy labels. Then, ε drag is introduced in label semantic embedding learning to learn more discriminative hash codes. In order to enhance the similarity semantic information in image and text hash codes, both intra-modal similarity and inter-modal similarity of images and texts are considered. A robust hashing unpaired zero-shot cross-modal retrieval model is established to solve the unpaired zero-shot cross-modal retrieval problem with label noise. The present application comprises the following steps:

[0006] Step 1, feature extraction of cross-modal data set: feature extraction of images and texts in the original cross-modal data set and attribute vector extraction of classes contained in the data set;

[0007] Step 2, cross-modal data set processing: according to the unpaired zero-shot cross-modal retrieval scenario with noisy labels, label noise is added to the existing cross-modal data set, unpaired processing is performed, and visible classes and invisible classes are divided;

[0008] Step 3, establishment of robust hashing unpaired zero-shot cross-modal retrieval model: first, sparse constraints and low-rank constraints are imposed on the noise matrix and the recovered label matrix, respectively, to effectively reduce the negative impact of noisy labels. Then, ε drag method is introduced in label semantic embedding learning to learn more discriminative hash codes. In addition, in order to enhance the similarity semantic information in image and text hash codes, both intra-modal similarity and inter-modal similarity are considered. Finally, a zero-shot cross-modal hashing retrieval framework is designed to learn different hash codes for different modalities, so that it is suitable for unpaired retrieval scenarios;

[0009] Step 4, update and optimization of unpaired zero-shot cross-modal retrieval model: an iterative discrete optimization algorithm is designed to solve the unpaired zero-shot cross-modal retrieval model obtained in Step 3 until the model converges or the maximum number of iterations is reached, obtaining the hash code of the image and text training set and the hash function;

[0010] Step 5, Unpaired zero-shot cross-modal retrieval of image-text: First, the hash codes of the retrieval set and the query set of the unpaired image-text cross-modal retrieval are obtained by the hash function trained in Step 4; then, the query set is put into the retrieval set for query, and the query result is obtained by calculating the Hamming distance between the hash codes of the query set and the hash codes of each sample in the retrieval set, and the one with the smallest Hamming distance is the final query result.

[0011] Further, the feature extraction method of the image, text and class attribute in Step 1 is as follows: the image features are extracted by VGG-16 model; the text features are extracted by bag-of-words model; and a corresponding word vector is extracted for each class name by Glove method to form a class attribute matrix.

[0012] Further, the image-text cross-modal dataset processing method in Step 2 is as follows: first, 10% of the samples in the original image-text cross-modal dataset are selected as the query set, and the remaining samples are used as the training set; second, 20% of all classes in the entire dataset are randomly selected as invisible classes, and the remaining classes are visible classes; third, in order to meet the zero-shot image-text cross-modal retrieval scenario, the training set, query set and retrieval set in the dataset are re-divided; the division method is to take the sample pairs corresponding to the invisible classes in the query set obtained in the first step as the new query set; take the sample pairs corresponding to the visible classes in the training set obtained in the first step as the new training set; and the retrieval set is composed of the training set obtained in the first step. Fourth, different proportions of label noise are added to the samples in the training set, and the specific method is to replace the "1" and "0" in the label, and at the same time, randomly shuffle the sample order of the image and text modal in the training set, so as to form an unpaired zero-shot image-text cross-modal retrieval scenario with noisy labels.

[0013] Further, in Step 2, in order to ensure the generalization ability of the model, the visible classes and invisible classes are randomly re-divided to obtain new training set and query set each time the model is trained, and at the same time, the newly obtained training set is re-labeled with noise and the sample order is shuffled. The average of multiple training results is taken as the final result.

[0014] Further, the specific steps of establishing a robust hash unpaired zero-shot image-text cross-modal retrieval model in Step 3 are as follows:

[0015] Step 3.1, Noise label learning: In order to alleviate the noise label problem in the image-text cross-modal training set, the observed label is decomposed into two parts: recovered label and noise. Due to the potential correlation between labels, the recovered label should exhibit low-rank characteristics, and a sparse constraint on the noise matrix is introduced to enhance the robustness of the algorithm, which is stated by formula (1):

[0016]

[0017] where, denotes the original noisy label matrix of the i-th modality, denotes the recovered label matrix of the i-th modality, D (i) corresponding to the noisy matrix of the i-th modality, including two modalities of image and text.

[0018] Step 3.2, semantic enhancement learning: in the hash code learning of image and text, in order to enhance the semantic discriminability of hash code, the label information will be embedded in the hash code. In addition, the different categories of regression targets are moved in the opposite direction by using ε drag, so as to expand the gap between different categories, so as to further enhance the discriminability in the hash code. Semantic enhancement learning can be obtained by minimizing the following problem:

[0019]

[0020] where, denotes the label matrix of the i-th modality after ε drag, C (i) and F (i) respectively denote the direction matrix and numerical matrix of the i-th modality after ε drag. H (i) denotes the orthogonal projection matrix of the i-th modality, B (i) denotes the hash code of the i-th modality, and k is the length of the hash code, is the number of training samples of the i-th modality.

[0021] Step 3.3, attribute space learning: the attribute space is constructed through the attribute relationship between classes to realize the knowledge transfer from visible class to invisible class. In addition, the recovered label is embedded into the attribute space, which not only can improve the semantic information of the attribute space, but also can use the data characteristics in the attribute space to guide the learning of the recovered label. The attribute space of image and text modalities is obtained by optimizing the following problem:

[0022]

[0023] where, X (i) denotes the feature matrix of the i-th modality, A is the class attribute matrix, P (i) denotes the projection matrix of the i-th modality.

[0024] Step3.4, Similarity preserving learning: In order to fully embed semantic information in the hash codes of images and texts, semantic similarity is used to further expand the distance between classes and reduce the distance within classes. Specifically, in the unpaired image-text cross-modal retrieval, not only the intra-modal similarity of images and texts should be focused on, but also the inter-modal similarity of images and texts. The inter-modal similarity can effectively solve the problem of heterogeneity difference between modalities and improve the performance of image-text cross-modal retrieval. Similarity preserving learning is performed by optimizing the following problem:

[0025]

[0026] where V (i) and V (j) represent the latent representation of the i-th modality and the j-th modality, respectively, S (ii) is the intra-modal similarity of the i-th modality, and S (ii) is constructed as follows: S (12) represents the inter-modal similarity of images and texts, and S (12) is constructed as follows: where m and η f are weight parameters.

[0027] Step3.5, Hash function learning of images and texts: The hash function of images and texts is learned through a least squares regression problem. However, using only linear transformation may not exhibit sufficient discrimination ability. The hash function is learned by adding drift in the linear transformation:

[0028]

[0029] where t (i) represents the drift of the i-th modality, and e (i) is an element 1 vector. W (i) is the projection matrix of the i-th modality.

[0030] Further, when the unpaired zero-sample image-text cross-modal retrieval model is established in Step3, there are both linear and nonlinear relationships in the features of different modalities. Linear models can only capture linear relationships in features. In order to capture nonlinear relationships in features, a kernel function is added to the feature matrix to expand it to a nonlinear model.

[0031] Further, the model constructed in Step 3 includes the following parts. First, sparse constraints and low-rank constraints are imposed on the noise matrix and the recovered label matrix respectively to effectively reduce the negative impact of noisy labels. Then, an epsilon dragging method is introduced in the label semantic embedding learning to learn more discriminative hash codes. In addition, in order to enhance the similarity semantic information in image and text hash codes, both intra-modal similarity and inter-modal similarity of image and text modalities are considered. Second, a zero-shot cross-modal image-text hash retrieval framework is designed to learn different hash codes for image and text modalities, which is suitable for the unpaired cross-modal image-text retrieval scenario. The objective function corresponding to the model is:

[0032]

[0033] wherein, represents the regularization term of the model, which is used to prevent overfitting; μ, θ, η, λ, α, β, μ1, ξ and γ are non-negative parameters, φ(X (i) ) is the matrix obtained by the kernel function of the i-th modality feature.

[0034] Further, the update optimization of the unpaired zero-shot cross-modal image-text retrieval model in Step 4 is updated based on the objective function corresponding to the model established in Step 3. Obviously, the objective function corresponding to the model is not optimal, and the function needs to be optimized. The objective function is a non-convex problem, which brings difficulty to optimization. However, when updating a matrix variable by fixing other variables, a non-convex problem is converted into a convex problem, which is convenient for iterative update. The matrix variable is updated by using the alternating iterative algorithm until the objective function converges or the maximum number of iterations is reached, and finally the optimal hash code and hash function of the unpaired zero-shot cross-modal image-text retrieval model are obtained.

[0035] Further, the target function optimization step is as follows:

[0036] In order to solve the optimization problem of formula (6), an effective alternating iterative algorithm is proposed. First, an auxiliary variable J (i) is introduced to replace Subsequently, the augmented Lagrangian function of the model is as follows:

[0037]

[0038] wherein δ and ρ are penalty parameters. K (i) and Z (i) are Lagrange multipliers.

[0039] B (i) -step: fixing other variables, formula (7) is simplified as about B(i) Problem:

[0040]

[0041] B (i) The closed-form solution for P (i) is:

[0042] B (i) = sgn(A (i) V (ii) ). (9)

[0043] P (i) -step: Fixing other variables, equation (7) simplifies to a problem of finding P (i) Problem:

[0044]

[0045] By setting the partial derivative of P (i) to zero, the closed-form solution for P (i) is:

[0046]

[0047] H (i) -step: Fixing other variables and removing terms irrelevant to H (i) , update H (i) as: (i)T H (i) = I constraint, the subproblem of finding H (i) can be simplified as:

[0048]

[0049] where Tr(·) denotes the trace norm. Equation (12) is a classical orthogonal Procrustes problem, which can be solved effectively using singular value decomposition (SVD). Specifically, let Q (i) be the SVD of Y (i) , i.e., Y (i) = Q (i) Σ (i)T Q (i) , where Y (i) and U (i) represent the transformation matrices. Let M (i) = U T H (i) Y (i) , thus equation (12) can be restated as:

[0050]

[0051] When M (i) = i, formula (13) reaches the maximum value, H (i) The optimal solution of H

[0052] H (i) = U (i) Y (i)T . (14)

[0053] W (i) -step: fixing other variables, formula (7) is simplified as the problem of W (i)

[0054]

[0055] By setting the partial derivative of W (i) to zero, the closed solution of W (i) is obtained as:

[0056] W (i) = (βV (i) -βt (i) e (i) )X (i)T (βX (i) X (i)T +γI) -1 . (16)

[0057] t (i) -step: fixing other variables, formula (7) is simplified as the problem of t (i)

[0058]

[0059] By setting the partial derivative of t (i) to zero, the closed solution of t (i) is obtained as:

[0060]

[0061] V (i) -step: fixing other variables, formula (7) is simplified as the problem of V (i)

[0062]

[0063]

[0064] According to problem (19) can be restated as follows:

[0065]

[0066] ​​​where

[0067] Solve V (i) .

[0068] Fixing other variables, formula (7) is simplified to the problem of :

[0069]

[0070] By setting the partial derivative of to zero, the solution of :

[0071]

[0072] In addition, the recovered label matrix is extended to Therefore, the update of :

[0073]

[0074] D (i) -step: Fixing other variables, formula (7) is simplified to the problem of (i) :

[0075]

[0076] Update D (i) according to the following rules:

[0077]

[0078] J (i) -step: Update J (i) by solving the following sub-problems:

[0079]

[0080] Z (i) ,K (i) ,ρ,δ-step: The Lagrange multipliers Z (i) ,K (i) and the penalty parameters ρ,δ are updated as follows:

[0081]

[0082] where ρ max is a predefined maximum number, and ε>1 is a constant.

[0083] Optimize formula (7) by the above steps until the function converges or reaches the maximum number of iterations, stop iteration, and get the optimal hash code and hash function of the image and text modal.

[0084] The beneficial effects of the present application are:

[0085] 1、The present application provides a robust hashing unpaired zero-shot cross-modal image-text retrieval method. It overcomes the following limitations of existing zero-shot cross-modal image-text retrieval methods: 1) Most existing methods assume that the labels of the training data are complete and accurate. However, the labels required in training are marked by artificial marking, so there will be often omissions and incorrect category assignments in the marking process. 2) Secondly, most zero-shot cross-modal hashing retrieval methods assume that all modal data are completely paired. However, in practical applications, different modal data often have the problems of loss and alignment error, so it is unrealistic to assume that different modal data are completely paired. 3) In addition, the existing zero-shot cross-modal hashing retrieval method does not fully utilize the discriminative ability of label information in the learning of label-guided hash code, thereby affecting the retrieval performance.

[0086] 2、Specifically, the present application designs a zero-shot cross-modal hashing retrieval framework to learn different hash codes for different modalities, making it suitable for unpaired retrieval scenarios. In addition, sparse constraints and low-rank constraints are imposed on the noise matrix and the recovered label matrix, respectively, to effectively reduce the negative impact of noisy labels. Then, epsilon dragging is introduced in the label semantic embedding learning to learn more discriminative hash codes. In order to enhance the similarity semantic information in image and text hash codes, while considering the intra-modal similarity and inter-modal similarity of images and texts, the present application optimizes the objective function through an effective alternating iteration algorithm, effectively avoiding quantization errors; the present application improves the retrieval accuracy in the unpaired zero-shot cross-modal image-text retrieval scenario with noisy labels, and has better performance. BRIEF DESCRIPTION OF DRAWINGS

[0087] The drawings accompanying the specification of the present application form part of the present application and serve to further understand the present application.

[0088] Figure 1 The figure is the architecture diagram of the embodiment method of the present application.

[0089] Figure 2 The figure is the flowchart of the iterative update of the method of the present application. DETAILED DESCRIPTION

[0090] The following description is exemplary and is intended to further illustrate the technical solutions of the present application in conjunction with the drawings.

[0091] Example 1

[0092] Figure 1 is a robust hashing of the invention of a non-paired zero sample graph-text cross-modal retrieval method architecture. Take graph-text cross-modal retrieval as an example.

[0093] In this example, referring to Figure 1 , the method of the present application specifically includes the following processes:

[0094] Step 1, feature extraction of graph-text cross-modal dataset: feature extraction of image and text in original graph-text cross-modal dataset and attribute vector extraction of classes contained in the dataset. In this example, the dataset used includes two modalities of image and text, and a one-to-one corresponding label. In Step 1, the class attribute is extracted, and a corresponding word vector is extracted for each class name using the Glove method to form a class attribute matrix.

[0095] Step 2, graph-text cross-modal dataset processing: since the problem to be solved by the present application is the non-paired zero sample graph-text cross-modal retrieval problem with label noise, the already processed graph-text cross-modal dataset cannot be directly used. According to the application scenario of non-paired zero sample graph-text cross-modal retrieval with label noise, the dataset should be reprocessed and divided to meet the application scenario of graph-text cross-modal retrieval. The specific processing method is as follows:

[0096] First, select 10% of the samples in the original graph-text cross-modal dataset as the query set, and the rest as the training set; second, randomly select 20% of all classes in the entire dataset as invisible classes, and the rest as visible classes; third, in order to meet the zero sample graph-text cross-modal retrieval scenario, the training set, query set and retrieval set in the dataset are re-divided; the division method is to take the sample pairs corresponding to the invisible classes in the query set obtained in the first step as the new query set; take the sample pairs corresponding to the visible classes in the training set obtained in the first step as the new training set; the retrieval set is composed of the training set obtained in the first step. Fourth, add different proportions of label noise to the samples in the training set, and randomly shuffle the order of samples in different modalities in the training set to form a non-paired zero sample graph-text cross-modal retrieval scenario with label noise.

[0097] Let represent a multi-modal dataset including two modalities of image and text. The feature matrix of the i-th modality is represented as where d i is the feature dimension, n i is the total number of samples. The label matrix is where c represents the total number of classes. In addition, the class attribute matrix is d AThe dimension of class attribute. In the zero-shot cross-modal retrieval task, the dataset is divided into two sets: the visible dataset and the invisible dataset The visible dataset of the i-th modality contains samples, while the invisible dataset contains samples. Similarly, the label matrix L (i) is divided into and and The following should be met:

[0098] Step 3, establish a robust hash of unpaired zero-shot cross-modal retrieval model: first, the noise matrix and the recovered label matrix are respectively imposed with sparse constraint and low rank constraint, so as to effectively reduce the negative impact of noise labels; then, the ε drag method is introduced in the label semantic embedding learning, so as to learn more discriminative hash code; in addition, in order to enhance the similarity semantic information in image and text hash code, both intra-modal similarity and inter-modal similarity are considered; finally, a zero-shot cross-modal hash retrieval framework is designed to learn different hash codes for different modalities, so as to be suitable for unpaired retrieval scene.

[0099] Next, the learning model of each module will be described in detail:

[0100] Step 3.1, noise label learning: in order to alleviate the problem of noise labels in cross-modal training set, the observed labels are decomposed into two parts: recovered labels and noise. Due to the potential correlation between labels, the recovered labels should exhibit low rank characteristics, and sparse constraint on noise matrix is introduced to enhance the robustness of the algorithm, which is stated by formula (28):

[0101]

[0102] Wherein, represents the original noise label matrix of the i-th modality, represents the recovered label matrix of the i-th modality, D (i) corresponds to the noise matrix of the i-th modality, including image and text two modalities.

[0103] Step 3.2, semantic enhancement learning: in the hash code learning of image and text, in order to enhance the semantic discriminability of hash code, the label information will be embedded in the hash code. In addition, the ε drag is used to make the regression targets of different categories move in opposite directions, so as to expand the gap between different categories, so as to further enhance the discriminability of hash code. Semantic enhancement learning can be obtained by minimizing the following problem:

[0104]

[0105] wherein, denotes the label matrix of the i-th modality after ε-dragging, C (i) and F (i) denote the direction matrix and the numerical matrix of the i-th modality after ε-dragging, respectively. H (i) denotes the orthogonal projection matrix of the i-th modality, B (i) denotes the hash code of the i-th modality, and k is the length of the hash code, is the number of training samples of the i-th modality.

[0106] Step 3.3, attribute space learning: the attribute space is constructed through the attribute relationship between classes to realize the knowledge transfer from visible classes to invisible classes. In addition, the recovered label is embedded in the attribute space, which not only improves the semantic information of the attribute space, but also uses the data features in the attribute space to guide the learning of the recovered label. The attribute spaces of the image and text modalities are obtained by optimizing the following problem:

[0107]

[0108] wherein, X (i) denotes the feature matrix of the i-th modality, A is the class attribute matrix, P (i) denotes the projection matrix of the i-th modality.

[0109] Step 3.4, similarity preserving learning: in order to fully embed semantic information in the hash codes of images and texts, semantic similarity is used to further expand the inter-class distance and reduce the intra-class distance. Specifically, in the unpaired image-text cross-modal retrieval, not only the intra-modal similarity but also the inter-modal similarity should be considered. The inter-modal similarity can effectively solve the problem of heterogeneity difference between modalities and improve the performance of image-text cross-modal retrieval. The similarity preserving learning is performed by optimizing the following problem:

[0110]

[0111] wherein, V (i) and V (j) denote the latent representation of the i-th modality and the j-th modality, respectively, S (ii) is the intra-modal similarity of the i-th modality, S (ii) is constructed as follows: S (12) denotes the inter-modal similarity of the image and text modalities, S (12) is constructed as follows: wherein m and η f are weight parameters.

[0112] Step3.5, Hash function learning of image and text: learn the hash function of image and text through the least square regression problem, however, only using linear transformation may not show enough strong discrimination ability. Learn the hash function by adding drift in the linear transformation:

[0113]

[0114] wherein, t (i) represents the drift of the i-th modality, and e (i) is the vector of element 1. W (i) is the projection matrix of the i-th modality.

[0115] Further, when the model is established in Step3, there are both linear and nonlinear relationships in the features of different modalities. The linear model can only capture the linear relationship in the features. In order to capture the nonlinear relationship in the features, a kernel function is added to the feature matrix to expand it into a nonlinear model.

[0116] Further, the model constructed in Step3 has the following parts. First, sparse constraint and low rank constraint are applied to the noise matrix and the recovered label matrix respectively to effectively reduce the negative impact of noisy labels. Then, the ε drag method is introduced in the label semantic embedding learning to learn more discriminative hash codes. In addition, in order to enhance the similarity semantic information in the hash codes, both intra-modality similarity and inter-modality similarity are considered. Second, a zero-shot cross-modal image-text hash retrieval framework is designed to learn different hash codes for different modalities, which is suitable for the unpaired cross-modal image-text retrieval scene. The objective function of the model is:

[0117]

[0118] wherein, represents the regularization term of the model, which is used to prevent overfitting; μ, θ, η, λ, α, β, μ1, ξ and γ are non-negative parameters, φ(X (i) ) is the matrix obtained by the kernel function of the i-th modality feature.

[0119] Step4, update optimization of unpaired zero-shot cross-modal image-text retrieval model: design an iterative discrete optimization algorithm to solve the unpaired zero-shot cross-modal image-text retrieval model obtained in Step3, until the model converges or reaches the maximum iteration number, and obtain the hash code of the training set and the hash function;

[0120] The updating and optimization of the model in Step 4 is the optimization of the model obtained in Step 3. Obviously, the objective function corresponding to the model is not optimal, and the function needs to be optimized. The objective function is a non-convex problem, which brings difficulties to optimization. However, when updating a matrix variable by fixing other variables, a non-convex problem is converted into a convex problem, which is convenient for iterative updating. An alternating iterative algorithm is used to update the matrix variable until the objective function converges or the maximum number of iterations is reached, and finally the optimal hash code and hash function are obtained.

[0121] The function (33) is not optimal, and needs to be iteratively updated. Obviously, the overall objective function is a non-convex optimization problem. Therefore, the present application proposes an effective iterative discrete optimization algorithm to solve this problem. Specifically, see Figure 2 The optimization step of formula (33) is as follows:

[0122] In order to solve the optimization problem of formula 33 objective function, an effective alternating iterative algorithm is proposed. First, introduce an auxiliary variable J (i) to replace Subsequently, the augmented Lagrangian function of the model is as follows:

[0123]

[0124] Where δ and ρ are penalty parameters. K (i) and Z (i) are Lagrange multipliers.

[0125] B (i) -step: fixing other variables, formula (34) is simplified to the problem of B (i) :

[0126]

[0127] The closed solution of B (i) is:

[0128] B (i) = sgn(λV (i) + θkV (i) S (ii) ). (36)

[0129] P (i) -step: fixing other variables, formula (34) is simplified to the problem of P (i) :

[0130]

[0131] By setting the partial derivative of P (i) to zero, P(i) The closed-form solution for H

[0132]

[0133] H (i) -step: Fixing other variables and removing items irrelevant to H (i) , update H (i) subject to H (i)T H (i) = I constraint, H (i) The subproblem can be simplified as:

[0134]

[0135] where Tr(·) denotes the trace norm. Equation (39) is the classical orthogonal Procrustes problem, which can be solved efficiently using singular value decomposition (SVD). Specifically, let Using singular value decomposition Q (i) , i.e., Q (i) ≈ Y (i) Σ (i) u (i)T , where Y (i) and u (i) represent the transformation matrices. Let M (i) = U (i) T H (i) Y (i) Thus, equation (39) can be restated as follows:

[0136]

[0137] When M (i) = I, formula (40) reaches the maximum, and the optimal solution for H (i) is as follows:

[0138] H (i) = U (i) Y (i)T . (41)

[0139] W (i) -step: Fixing other variables, formula (34) is simplified to a problem about W (i) :

[0140]

[0141] By setting the partial derivative of W (i) to zero, the closed-form solution for W (i) is as follows:

[0142] W (i)= (βV (i) - βt (i) e (i) )X (i)T (βX (i) L (i)T + γI) -1 . (43)

[0143] t (i) -step: Fixing other variables, formula (34) is simplified to the problem of t (i) :

[0144]

[0145] By setting the partial derivative of t (i) to zero, the closed solution of t (i) is obtained as:

[0146]

[0147] V (i) -step: Fixing other variables, formula (34) is simplified to the problem of V (i) :

[0148]

[0149] According to the problem (46), it can be restated as follows:

[0150]

[0151] where The approximate maximum algorithm is used to solve V (i) .

[0152] Fixing other variables, formula (34) is simplified to the problem of :

[0153]

[0154] By setting the partial derivative of to zero, the solution of can be expressed as:

[0155]

[0156] In addition, the label matrix restored by introducing ε drag is extended to Therefore, the update of is as follows:

[0157]

[0158] D (i) -step: Fixing other variables, formula (34) is simplified as about D (i) Problem:

[0159]

[0160] According to the following rules, update D (i) :

[0161]

[0162] J (i) -step: Update J by solving the following sub-problems (i) :

[0163]

[0164] Z (i) ,K (i) ,ρ,δ-step: Lagrange multiplier Z (i) ,K (i) and penalty parameters ρ,δ are updated as follows:

[0165]

[0166] where ρ max is a predefined maximum number, and ε>1 is a constant.

[0167] Step 5, robust hashing of unpaired zero sample cross-modal retrieval of text and image: first, the hash code of the retrieval set and the query set is obtained through the hash function trained in Step 4; then, the query set is put into the retrieval set for query, and the query result is obtained by calculating the Hamming distance between the hash code of the query set and the hash code of each sample in the retrieval set, and the one with the smallest Hamming distance is the final query result. The specific implementation steps are as follows:

[0168] Given the feature matrix corresponding to the query sample pair of image and text as φ(X query (1) ) and φ(X query (2) ), combine the projection matrix W (1) and W (2) obtained in model training and the drift t (1) and t (2) . Through the formula B x =sgn(W (1) φ(X query (1) )+t (1) e (1) )) and B y= sgn(W (2) φ(X query (2) + t (2) e (2) )) to obtain the hash code corresponding to the query sample.

[0169] Since the query task of the application is performed in a binary space, the result of the query is obtained by calculating the Hamming distance between the query sample and each sample in the retrieval set. The sample corresponding to the minimum Hamming distance in the retrieval set is the query result obtained.

[0170] In order to illustrate the effect of the application, the technical solutions of the application are further described below through specific embodiments:

[0171] 1. Simulation conditions

[0172] The application is simulated by using Matlab software. The experiment is performed on the cross-modal data set Wiki (containing two modalities of images and texts). The experiment includes two query tasks: (1) text query image (Text2Img), (2) image query text (Img2Text).

[0173] 2. Simulation content

[0174] The method of the present application is an unpaired robust hashing with noisy labels (URHNL) method. The method is compared with existing non-noisy sample cross-modal hashing retrieval methods and zero sample cross-modal hashing retrieval methods. The non-noisy sample cross-modal hashing retrieval methods include: (1) label consistent flexible matrix factorization hashing (LFMH), (2) discrete latent factor hashing (DLFH), (3) adaptive marginalized semantic hashing (AMSH), (4) supervised adaptive partial multi-view hashing (SAPMH), (5) enhanced discrete multi-modal hashing (EDMH), and (6) matrix tri-factorization hashing (MTFH). The zero sample single-modal hashing retrieval methods include: (1) attribute hashing (AH) for zero sample image retrieval, (2) zero-shot hashing with orthographic projection (HOP), and (3) zero-shot hashing via transferring supervised knowledge (TSK). The zero sample cross-modal hashing retrieval method includes: (1) discrete asymmetric zero-shot hashing (DAZSH). For the zero sample single-modal hashing retrieval method, the hash codes of the image and text modalities are obtained by a single-modal model, and then the following query task is performed.

[0175] 3. Simulation results

[0176] The simulation experiment respectively gives the experimental results of the comparative method and the method proposed in the application under the data set Wiki. In order to meet the zero sample cross-modal retrieval scene of text and image, the application randomly selects 20% of the classes in the data set Wiki as invisible classes. In the data set Wiki, there are a total of 10 classes, according to the experimental setting, 2 classes are randomly selected as invisible classes, and the processing mode of the remaining data set is the same as that of the application. For label noise, the noise ratio (LNR) is set to 0.2, and noise labels are introduced by replacing a certain percentage of "1" and "0" labels. For the unpaired retrieval scene, the samples in the two modalities of push and text are randomly shuffled to simulate.

[0177] In this simulation experiment, a widely used indicator is used to measure the performance of the URHNL method proposed in the application and other comparative methods. That is, the average value of the average precision (mAP). Given a query and a retrieval result list, the average precision (AP) is defined as:

[0178]

[0179] Where N is the number of relevant instances in the retrieval set, P(r) is defined as the precision of the rth retrieval instance, and if the rth retrieval instance is the true neighbor of the query, then δ(r)=1; otherwise δ(r)=0. Then, AP of all queries is averaged to obtain mAP. The evaluation rule is that the larger the mAP value, the better the performance.

[0180] The hash code length taken from the simulation experiment is 24 bits, 32 bits, 64 bits and 128 bits, and the mAP values corresponding to the URHNL method proposed in the application and other comparative methods are shown in Tables 1 and 2.

[0181] Table 1 mAP values of different methods on Wiki data set Text2Img (LNR=0.2)

[0182]

[0183] Table 2 mAP values of different methods on Wiki data set Text2Img (LNR=0.2)

[0184]

[0185] From Tables 1 and 2, it can be seen that the mAP values of the URHNL method proposed in the application in the two query tasks in the unpaired zero sample cross-modal retrieval scene of text and image carrying label noise in the Wiki data set are higher than those of other comparative methods. Further proving the superiority of the URHNL method proposed in the application in zero sample cross-modal retrieval.

[0186] The above embodiments only express the specific implementation of the present application, which is described in more detail and in more detail, but cannot be understood as a limitation on the scope of the patent of the present application. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application.

Claims

1. A robust hashing unpaired zero-shot cross-modal retrieval method for text and images, characterized in that: The specific steps of the method are as follows: Step 1, feature extraction of image-text cross-modal data set: feature extraction of image and text in original image-text cross-modal data set and attribute vector extraction of classes contained in the data set; Step 2, image-text cross-modal data set processing: according to the unpaired zero sample image-text cross-modal retrieval scene with noisy labels, add label noise to the existing image-text cross-modal data set, perform unpaired processing, and divide visible classes and invisible classes; Step 3, establishment of robust hash unpaired zero sample image-text cross-modal retrieval model: first, impose sparse constraint and low rank constraint on noise matrix and recovered label matrix respectively to effectively reduce the negative influence of noisy labels; then, introduce epsilon dragging method in label semantic embedding learning to learn more discriminative hash code; in addition, in order to enhance the similarity semantic information in image and text hash code, the intra-modal similarity and inter-modal similarity of image and text are considered; finally, a zero sample image-text cross-modal hash retrieval framework is designed to learn different hash codes for image and text modal at the same time, so as to adapt to unpaired retrieval scene; Step 4, update and optimization of unpaired zero sample image-text cross-modal retrieval model: design iterative discrete optimization algorithm to solve the unpaired zero sample image-text cross-modal retrieval model obtained in Step 3 until the model converges or reaches the maximum iteration number, and obtain the hash code of the training set and the hash function; Step 5, unpaired zero sample image-text cross-modal retrieval: first, obtain the hash code of the retrieval set and the query set of the unpaired image-text cross-modal retrieval through the hash function trained in Step 4; then, put the query set into the retrieval set for query, and obtain the query result by calculating the hamming distance between the hash code of the query set and the hash code of each sample in the retrieval set, and the one with the smallest hamming distance is the final query result; The Step 3 comprises: Step 3.1, noise label learning: in order to alleviate the noise label problem in image-text cross-modal training set, the observed label is decomposed into two parts: recovered label and noise; due to the potential correlation between labels, the recovered label should show low rank characteristics, and sparse constraint on noise matrix is introduced to enhance the robustness of the algorithm, which is stated by formula (1): （1） wherein, represents the original noise label matrix of the th modality, represents the recovered label matrix of the th modality, corresponds to the noise matrix of the th modality, and there are two modalities including images and texts. Step 3.2, semantic enhancement learning: in the learning of image and text hash code, label information is embedded in the hash code to enhance the semantic discriminability of the hash code; in addition, epsilon dragging is used to make the regression targets of different categories move in opposite directions, so as to enlarge the gap between different categories, so as to further enhance the discriminability in the hash code; semantic enhancement learning is obtained by minimizing the following problem: ； （2） wherein, denotes the label matrix of the th modality after ε-dragging, and denote the direction matrix and the numerical matrix of the th modality after ε-dragging, respectively, denotes the orthogonal projection matrix of the th modality, denotes the hash code of the th modality, is the hash code length, is the number of training samples of the th modality; Step 3.3, attribute space learning: the attribute space is constructed through the attribute relationship between classes to realize the knowledge transfer from visible class to invisible class; in addition, the recovered label is embedded into the attribute space, which not only improves the semantic information of the attribute space, but also uses the data characteristics in the attribute space to guide the learning of the recovered label; the attribute space of image and text modal is obtained by optimizing the following problem: （3） wherein, represents a feature matrix of the th modality, is a class attribute matrix, represents a projection matrix of the th modality; Step3.4, Similarity preserving learning: In order to fully embed semantic information in image and text hash codes, semantic similarity is used to further expand the distance between classes and reduce the distance within classes; Specifically, in the unpaired image-text cross-modal retrieval, not only the intra-modal similarity of image and text should be focused on, but also the inter-modal similarity of image and text should be focused on, the inter-modal similarity can effectively solve the problem of heterogeneity difference between modalities, and improve the performance of image-text cross-modal retrieval, and similarity preserving learning is performed by optimizing the following problem: ； （4） wherein, and denote the latent representation of the th modality and the th modality, respectively, is the intra-modality similarity of the th modality, denotes the inter-modality similarity between image and text. Step3.5, Image and text hash function learning: The hash function of image and text is learned through the least squares regression problem, however, only using linear transformation cannot show enough strong discrimination ability, and the hash function is learned by adding drift in the linear transformation: （5） wherein, represents the drift of the th modality, while is the vector of element 1, is the projection matrix of the th modality; The objective function corresponding to the model constructed in Step3 is: ； （6） wherein, , denotes a regularization term of the model, for preventing overfitting; , , , and are non-negative parameters, is the matrix of the features of the th modality, obtained through a kernel function.

2. The method of claim 1, wherein the robust hashing of unpaired zero-shot cross-modal retrieval of text and images is characterized by: In Step1, the Glove method is used to extract a corresponding word vector for each class name to form a class attribute matrix.

3. The method of claim 1, wherein the method is a robust hashing unpaired zero-shot cross-modal retrieval method for text and image. The specific method of Step2 is: first, select 10% of the samples in the original image-text cross-modal data set as the query set, and the rest as the training set; Second, randomly select 20% of all classes in the entire data set as invisible classes, and the rest as visible classes; Third, in order to meet the zero-shot image-text cross-modal retrieval scenario, the training set, query set and retrieval set in the data set are re-divided; The division method is to take the sample pairs corresponding to the invisible classes in the query set obtained in the first step as the new query set; Take the sample pairs corresponding to the visible classes in the training set obtained in the first step as the new training set; The retrieval set is composed of the training set obtained in the first step; Fourth, add different proportions of label noise to the samples in the training set, and randomly shuffle the order of samples in different modalities in the training set, so as to form an unpaired zero-shot image-text cross-modal retrieval scenario with noisy labels.

4. The method of claim 1, wherein the robust hashing of unpaired zero-shot cross-modal retrieval of text and images is characterized by: In Step4, the matrix variable is updated by using the alternating iterative algorithm until the objective function converges or the maximum number of iterations is reached, and finally the optimal hash codes and hash functions of image and text are obtained.

Images