Label noise-oriented semi-supervised low-rank cross-modal hash retrieval method and system

Abstract

The invention discloses a semi-supervised low-rank cross-modal Hash retrieval method and a semi-supervised low-rank cross-modal Hash retrieval system for label noise. The method comprises the following steps: firstly, constructing an intra-modal and inter-modal contrast learning loss function, a structured semantic similarity loss function oriented to all samples, a cross-modal semantic correlation loss function and a label noise-oriented low-rank learning loss function, then constructing a total objective function, and respectively learning projection matrixes for a first modal and a second modal; based on the projection matrix, binary hash codes are generated for samples in the query sample set and the retrieval sample set; hamming distances from query samples to all samples in a retrieval sample set are calculated based on binary hash coding, samples corresponding to the first Z minimum Hamming distances are taken as retrieval results, and the problems that in the prior art, when training sample data are affected by label noise, effective supervision information is reduced, and the retrieval efficiency is improved are solved. Therefore, the technical problem that the trained cross-modal retrieval model cannot well mine identification features is solved, and the cross-modal retrieval precision oriented to the label noise influence is improved.

Description

Technical Field

[0001] The present invention relates to the field of cross-modal retrieval technology, and in particular to a semi-supervised low-rank cross-modal hash retrieval method and system for label noise. Background Art

[0002] With the development of technology and the expansion of application scenarios, in today's big data era, the amount of data is growing exponentially, and heterogeneous data from modalities such as images and text modalities are also growing rapidly. Data valueization is an important cornerstone for adjusting production relations and driving productivity revolution in the digital age, and it helps promote the creation, transfer, transformation and amplification of data value. Mining valuable information from massive big data in modalities such as images and texts, and realizing the value of image big data and text big data can fully release the economic value, social value and governance value of data. Efficient cross-modal retrieval methods are the key link and urgent need for data valueization.

[0003] Cross-modal retrieval aims to retrieve semantically similar samples in another modality based on samples of one modality. For cross-modal retrieval methods, increasing and utilizing the supervised information of modal samples such as images and texts is conducive to digging deep into the connections between samples of different modalities, but labeling samples to obtain supervisory information often requires huge costs, which makes the number of labeled samples very limited. On the basis of a small number of labeled samples, making full and effective use of a large number of unlabeled samples can significantly improve the accuracy of cross-modal retrieval. Researching semi-supervised cross-modal retrieval methods for the influence of label noise has become an important way to obtain valuable information from modal big data such as images and texts. In the actual application scenarios of big data, the label information annotated for samples is sometimes wrong due to human or non-human factors, that is, there is label noise, which has an adverse effect on the training process of the cross-modal retrieval model, and then leads to a decrease in cross-modal retrieval performance. Therefore, designing a semi-supervised cross-modal retrieval method that considers the influence of label noise is crucial to meet the needs of actual application scenarios of big data. At the same time, it has important practical significance and wide application value for successfully completing cross-modal retrieval tasks in big data contexts. Summary of the invention

[0004] The present invention provides a semi-supervised low-rank cross-modal hash retrieval method and system for label noise, thereby solving the technical problem in the prior art that when the training sample data is affected by label noise, the effective supervision information is reduced, resulting in the trained cross-modal retrieval model being unable to well mine identification features, thereby achieving the technical effect of improving the accuracy of cross-modal retrieval under the influence of label noise.

[0005] The present invention provides a semi-supervised low-rank cross-modal hash retrieval method for label noise, comprising:

[0006] Constructing inter-modality contrast loss function Among them, f1(x 1j ) is the sample x of the first mode 1j The hash function, f2(x 2j ) is the sample x of the second mode 2j The hash function, f1(x 1k ) is the sample x of the first mode 1k The hash function, f2(x 2k ) is the sample x of the second mode 2k Hash function of

[0007] Construct the sample x of the first mode 1j With the enhanced sample x′ 1j Intra-modal contrastive loss function Among them, f1(x′ 1j ) is the sample x′ of the first mode 1j The hash function, f1(x′ 1k ) is the sample x′ of the first mode 1k Hash function of

[0008] Construct the sample x of the second mode 2j With the enhanced sample x′ 2j Intra-modal contrastive loss function Among them, f2(x′ 2j ) is the sample x′ of the second mode 2j The hash function, f2(x′ 2k ) is the sample x′ of the second mode 2k Hash function of

[0009] Constructing the total loss function for intra-modal and inter-modal contrastive learning Among them, λ1 and λ2 are hyperparameters;

[0010] By formula and Calculate the samples x of the first mode respectively 1i and x 1j The cosine similarity between and the sample x of the second mode 2i and x 2j The cosine similarity between

[0011] By formula and Respectively and Mapping is performed to obtain the mapped and

[0012] By formula Calculate the fusion similarity between paired samples of the two modalities

[0013] By formula Calculate the re-weighted fusion similarity Among them, NN t (i) represents the set of the t most recent sample pairs before the i-th sample pair;

[0014] By formula Calculate the structural similarity

[0015] By formula Calculate the structural semantic similarity Among them, ζ∈[0,1] is used to balance the fusion similarity and structural similarity Parameters;

[0016] Constructing a structured semantic similarity loss function Among them, K is the length of the hash code, f1(x 1p ) is the sample x of the first mode 1p The hash function, f2(x 2q ) is the sample x of the second mode 2q The hash function of For the matrix Elements in

[0017] Define the neighbor affinity matrix A used to capture the local information between different modalities. The element A of the matrix A is ij for: in, represents the k nearest neighbors of x, is the i-th labeled sample of the first modality and the jth labeled sample of the second modality The cross-modal similarity between is the mathematical expectation of the distance of all data pairs of the first mode, and the heat kernel parameter is the mathematical expectation of the distance of all data pairs of the second mode;

[0018] Define the inter-class similarity matrix S b , the matrix S b Elements in for: Define the intra-class similarity matrix S w , the matrix S w Elements in for: Among them, n cis the number of samples in the cth category, is the i-th column vector in the label matrix of the labeled samples of the first modality, is a column vector The cth element of is the jth column vector in the label matrix of the labeled samples of the second modality, is a column vector the cth element of ;

[0019] By the formula S = S w -S b Calculate the semantic similarity matrix S, the element S in S ij for:

[0020] Constructing a cross-modal semantic relevance loss function in, is the i-th labeled sample of the first modality The hash code of is the jth labeled sample of the second modality The hash code of

[0021] Constructing a low-rank learning loss function for label noise Among them, ||·|| * represents the nuclear norm of the matrix, L i is the label matrix containing noisy labels, is the ideal label matrix with labeled samples in the ith modality, is the feature matrix of the labeled samples of the i-th modality, and W is the matrix that maps the labeled samples of each modality to the ideal label matrix. The projection matrix, α and β are balance factors;

[0022] Constructing the overall objective function Learn projection matrices W1 and W2 for the first and second modalities respectively;

[0023] By formula Calculate the binary hash code of the query sample of the first modality By formula Calculate the binary hash code of the query sample of the second modality By formula Calculate the binary hash code of the jth sample in the retrieval sample set of the first modality By formula Calculate the binary hash code of the jth sample in the retrieval sample set of the second modality in, sgn(·) is the sign function, is the feature vector of the query sample of the first modality, is the feature vector of the query sample of the second modality, is the feature vector of the jth sample in the retrieval sample set of the first modality, is the feature vector of the jth sample in the retrieval sample set of the second modality;

[0024] By formula Calculate the Hamming distance between the query sample of the first modality and the jth sample in the retrieval sample set of the second modality By formula Calculate the Hamming distance between the query sample of the second modality and the jth sample in the retrieval sample set of the first modality

[0025] Take the first Z minimum Hamming distances in the retrieval sample set of the second modality The corresponding samples are taken as the retrieval results, and the first Z minimum Hamming distances are taken from the retrieval sample set of the first modality The corresponding samples are taken as retrieval results.

[0026] Specifically, the overall objective function is constructed The projection matrices W1 and W2 are learned for the first and second modalities respectively, including:

[0027] Fix W, W1 and W2 in the overall objective function and solve Fix the total objective function W1 and W2, solve W; fix the total objective function W and W2, solve W1; fix the total objective function W and W1, solve W2; until the preset number of iterations is reached, the final projection matrices W1 and W2 are obtained.

[0028] The present invention also provides a semi-supervised low-rank cross-modal hash retrieval system for label noise, comprising:

[0029] Inter-modal contrast loss function construction module, used to construct inter-modal contrast loss function Among them, f1(x 1j ) is the sample x of the first mode 1j The hash function, f2(x 2j ) is the sample x of the second mode 2j The hash function, f1(x 1k ) is the sample x of the first mode 1k The hash function, f2(x 2k ) is the sample x of the second mode 2k Hash function of

[0030] The contrast loss function construction module in the first modality is used to construct the sample x of the first modality 1j With the enhanced sample x′ 1j Intra-modal contrastive loss function Among them, f1(x′ 1j ) is the sample x′ of the first mode 1j The hash function, f1(x′ 1k ) is the sample x′ of the first mode 1k Hash function of

[0031] The contrast loss function building module in the second modality is used to construct the sample x of the second modality 2j With the enhanced sample x′ 2j Intra-modal contrastive loss function Among them, f2(x′ 2j ) is the sample x′ of the second mode 2j The hash function, f2(x′ 2k ) is the sample x′ of the second mode 2k Hash function of

[0032] The module for building the total loss function of intra-modal and inter-modal contrastive learning is used to build the total loss function of intra-modal and inter-modal contrastive learning. Among them, λ1 and λ2 are hyperparameters;

[0033] The cosine similarity calculation module is used to calculate the cosine similarity through the formula and Calculate the samples x of the first mode respectively 1i and x 1j The cosine similarity between and the sample x of the second mode 2i and x 2j The cosine similarity between

[0034] Mapping module, used to pass formula and Respectively and Mapping is performed to obtain the mapped and

[0035] Fusion similarity calculation module, used to calculate the formula Calculate the fusion similarity between paired samples of the two modalities

[0036] The re-weighted fusion similarity calculation module is used to calculate the similarity through the formula Calculate the re-weighted fusion similarity Among them, NNt (i) represents the set of the t most recent sample pairs before the i-th sample pair;

[0037] The structural similarity calculation module is used to calculate the similarity of Calculate the structural similarity

[0038] The structured semantic similarity calculation module is used to calculate the structured semantic similarity through the formula Calculate the structural semantic similarity Among them, ζ∈[0,1] is used to balance the fusion similarity and structural similarity Parameters;

[0039] Structured semantic similarity loss function building module, used to build structured semantic similarity loss function Among them, K is the length of the hash code, f1(x 1p ) is the sample x of the first mode 1p The hash function, f2(x 2q ) is the sample x of the second mode 2q The hash function of For the matrix Elements in

[0040] The neighbor affinity matrix building module is used to define the neighbor affinity matrix A for capturing the local information between different modalities. The element A of the matrix A ij for: in, represents the k nearest neighbors of x, is the i-th labeled sample of the first modality and the jth labeled sample of the second modality The cross-modal similarity between is the mathematical expectation of the distance of all data pairs of the first mode, and the heat kernel parameter is the mathematical expectation of the distance of all data pairs of the second mode;

[0041] Similarity matrix building module, used to define the inter-class similarity matrix S b , the matrix S b Elements in for: Define the intra-class similarity matrix S w , the matrix S w Elements in for: Among them, n c is the number of samples in the cth category, is the i-th column vector in the label matrix of the labeled samples of the first modality, is a column vector The cth element of is the jth column vector in the label matrix of the labeled samples of the second modality, is a column vector the cth element of ;

[0042] The semantic similarity matrix calculation module is used to calculate the semantic similarity matrix through the formula S = S w -S b Calculate the semantic similarity matrix S, the element S in S ij for:

[0043] Cross-modal semantic relevance loss function construction module, used to construct cross-modal semantic relevance loss function in, is the i-th labeled sample of the first modality The hash code of is the jth labeled sample of the second modality The hash code of

[0044] A module for building low-rank learning loss functions for label noise, which is used to build low-rank learning loss functions for label noise Among them, ||·|| * represents the nuclear norm of the matrix, L i is the label matrix containing noisy labels, is the ideal label matrix with labeled samples in the ith modality, is the feature matrix of the labeled samples of the i-th modality, and W is the matrix that maps the labeled samples of each modality to the ideal label matrix. The projection matrix, α and β are balance factors;

[0045] The overall objective function building module is used to build the overall objective function Learn projection matrices W1 and W2 for the first and second modalities respectively;

[0046] The hash code calculation module is used to calculate the Calculate the binary hash code of the query sample of the first modality By formula Calculate the binary hash code of the query sample of the second modality By formula Calculate the binary hash code of the jth sample in the retrieval sample set of the first modality By formula Calculate the binary hash code of the jth sample in the retrieval sample set of the second modality in, sgn(·) is the sign function, is the feature vector of the query sample of the first modality, is the feature vector of the query sample of the second modality, is the feature vector of the jth sample in the retrieval sample set of the first modality, is the feature vector of the jth sample in the retrieval sample set of the second modality;

[0047] The Hamming distance calculation module is used to calculate the distance between Calculate the Hamming distance between the query sample of the first modality and the jth sample in the retrieval sample set of the second modality By formula Calculate the Hamming distance between the query sample of the second modality and the jth sample in the retrieval sample set of the first modality

[0048] The retrieval module is used to obtain the first Z minimum Hamming distances in the retrieval sample set of the second modality. The corresponding samples are taken as the retrieval results, and the first Z minimum Hamming distances are taken from the retrieval sample set of the first modality The corresponding samples are taken as retrieval results.

[0049] Specifically, the overall objective function construction module is specifically used to construct the overall objective function Fix W, W1 and W2 in the overall objective function and solve Fix the total objective function W1 and W2, solve W; fix the total objective function W and W2, solve W1; fix the total objective function W and W1, solve W2; until the preset number of iterations is reached, the final projection matrices W1 and W2 are obtained.

[0050] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0051] First, the contrast learning loss function within and between modalities, the structured semantic similarity loss function for all samples, the cross-modal semantic relevance loss function and the low-rank learning loss function for label noise are constructed, and then the overall objective function is constructed to learn the projection matrices W1 and W2 for the first modality and the second modality respectively; based on the learned projection matrices W1 and W2 of the first modality and the second modality, binary hash codes are generated for the query sample and the samples in the retrieval sample set; based on the generated binary hash codes, the Hamming distance from the query sample to each sample in the retrieval sample set is calculated; and the samples corresponding to the first Z minimum Hamming distances in the retrieval sample set are taken as the cross-modal retrieval results. The present invention can improve the ability of cross-modal retrieval to identify similar samples in each modality by ensuring that the ideal label matrix of each modality is as similar as possible to the observed label matrix of labeled samples in each modality, thereby weakening the negative impact of label noise and improving the accuracy of cross-modal retrieval facing the influence of label noise. The present invention makes full use of labeled samples and unlabeled samples to enhance the association between modalities, promote the learning of intra-modal identification features, and improve the robustness of cross-modal retrieval. In addition, the preservation of cross-modal local relations used in the present invention can not only effectively mine the correlation information between different modalities, but also improve the temporal performance of cross-modal retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of a semi-supervised low-rank cross-modal hash retrieval method and system for label noise provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The embodiments of the present invention provide a semi-supervised low-rank cross-modal hash retrieval method and system for label noise, thereby solving the technical problem in the prior art that when the training sample data is affected by label noise, the effective supervision information is reduced, resulting in the trained cross-modal retrieval model being unable to well mine identification features, thereby achieving the technical effect of improving the accuracy of cross-modal retrieval under the influence of label noise.

[0054] The technical solution in the embodiment of the present invention is to solve the above technical problems, and the overall idea is as follows:

[0055] Assumptions is a multimodal dataset with C true categories and noisy labels, where M represents the number of modalities; is a dataset containing n labeled samples of the ith modality, where represents the jth labeled sample of the ith modality, d i is the dimension of the i-th modal feature, n is the number of labeled samples, The superscript L in indicates that the sample has a label. The embodiment of the present invention assumes that each modality is separately labeled by different annotators, and the label of the sample may be incorrect, that is, there is a noisy label; Represents the feature matrix of n labeled samples of the i-th modality; is a dataset containing m unlabeled samples of the ith modality, where represents the jth unlabeled sample of the ith modality, m is the number of unlabeled samples, The superscript U in indicates that the sample has no label; Represents the feature matrix of m unlabeled samples of the i-th modality; is a dataset containing n+m samples of the ith modality; for convenience, without considering the sample labels, let At the same time represents the feature matrix of n+m samples of the ith modality. Without affecting understanding, X i It also represents the feature matrix composed of deep learning features; L i ∈{0,1} C×n is the label matrix of the labeled samples of the i-th modality; For L i The pth column vector of The qth element of If the pth sample of the ith mode belongs to the qth class, then otherwise In the deep learning scenario, It also represents a minibatch dataset. The cross-modal retrieval in the embodiment of the present invention is a cross-modal retrieval for image modality and text modality, that is, the number of modalities M=2. For ease of understanding, the embodiment of the present invention stipulates that the image modality is the first modality and the text modality is the second modality. The method includes the following steps:

[0056] (1) Construct intra-modal and inter-modal contrastive learning loss functions;

[0057] Specifically, in order to better learn features, we consider inter-modal and intra-modal contrast losses. The inter-modal contrast loss is used to map the features of the two modalities to a common space, and the intra-modal contrast loss is used for feature mapping of each modality. Assume f1(x 1i )=sgn(W1 T x 1i )and are the hash functions of the first and second modes, respectively, where sgn(·) is the sign function, W1 and W2 are projection matrices, and W1 T and are the transposed matrices of matrices W1 and W2 respectively. The inter-modality contrast loss is designed as:

[0058]

[0059] Among them, S(u,v)=exp(cos(u,v)τ), cos(u,v)=u T v||u||2||v||2 is the cosine similarity, τ is the temperature parameter of the normalized temperature scaled cross entropy, and Q is the batch size. It can be seen that the inter-modality contrast loss The loss is calculated between paired samples.

[0060] The intra-modal contrast loss of the first modality is designed as sample x 1j With the enhanced sample x1′ j The contrast loss is:

[0061]

[0062] The intra-modal contrast loss for the second modality is designed as sample x 2j With the enhanced sample x2′ j The contrast loss is:

[0063]

[0064] In summary, the total contrast loss is:

[0065]

[0066] Among them, λ1 and λ2 are hyperparameters.

[0067] (2) Construct a structured semantic similarity preserving loss function for all samples;

[0068] Specifically, first calculate the cosine similarity between sample pairs in the two modalities and and The semantic similarity between sample pairs within each modality can be captured. and The distribution shape is close to the skewed distribution. and Map from the [-1,1] interval to the [0,1] interval, and use and Respectively represent the mapped and and It can be used to describe the probability that two samples in each modality are semantically similar.

[0069] Furthermore, the fusion similarity between paired samples of different modalities is calculated And map it to the interval [0,1]. When the value of fusion similarity is small, its reliability as a similarity measure is relatively low. For this problem, we can calculate a reweighted Solve. Use Reweighted Elements The specific definition of is:

[0070]

[0071] Among them, NN t (i) represents the set of the t closest sample pairs to the i-th sample pair.

[0072] Further, the structural similarity is defined as Based on the above definition, by simultaneously considering the fusion similarity between pairs of samples of different modalities and structural similarity Define structured semantic similarity as Among them, ζ∈[0,1] is used to balance the fusion similarity and structural similarity It can be seen that the structural semantic similarity It is a multi-level semantic similarity that can fully represent structured semantic information. Combining the structured semantic similarity, the following loss function can be designed:

[0073]

[0074] in, For the matrix elements. is the structural semantic similarity, specifically Among them, ζ∈[0,1] is used to balance the fusion similarity and structural similarity Parameters.

[0075] (3) Constructing a cross-modal local relationship preservation loss function;

[0076] Specifically, the high-level semantic description of an object often deviates from the low-level feature description. Therefore, completing the retrieval task based on the similarity between the original features cannot produce satisfactory results. Combining label information to maintain the local relationship between samples of different modalities is very important for mining semantic information and enhancing the relevance between different modalities. First, define the neighbor affinity matrix A for capturing the local information between different modalities. The elements of the matrix A are ij The specific definition of is:

[0077]

[0078] in, represents the k nearest neighbors of x, for and The cross-modal similarity between is the mathematical expectation of the distance of all data pairs of the first mode, and the heat kernel parameter is the mathematical expectation of the distance of all data pairs of the second mode.

[0079] Inspired by linear discriminant analysis, using S b and S w Respectively represent the inter-class and intra-class similarity matrices, matrix S b and S w Elements and is defined as follows:

[0080]

[0081] Among them, n c is the number of samples in the cth category. The cross-modal semantic relevance can be maximized by the following two criteria:

[0082]

[0083] in, and They are and is the hash code of , sgn(·) is the sign function, and W1 and W2 are projection matrices.

[0084] The criterion for maximizing cross-modal semantic relevance in formulas (10) and (11) can be rewritten as follows:

[0085]

[0086] in, is the cross-modal semantic relevance loss function, S = S w -S b is a unified semantic similarity matrix, the element S of S ij for:

[0087]

[0088] A ij are the elements of the neighbor affinity matrix A used to capture the local information between different modalities.

[0089] (4) Construct a low-rank learning loss function for label noise;

[0090] Specifically, when using labeled samples to train the model, label noise will reduce the ability of the cross-modal retrieval model to identify similar samples in each modality. According to the low-rank matrix recovery theory, a complete low-rank matrix can be recovered from an incomplete low-rank matrix containing unknown elements. Assume that the ideal label matrix of labeled samples in the i-th modality is In the presence of label noise, the label matrix L containing the noisy labels i It can be considered as a matrix with unknown elements. The ideal label matrix is It can be considered as the complete matrix that has been restored.

[0091] Assuming that there is a correlation between class labels in the ideal label matrix, the more similar the two class labels are, the more similar the sample sets belonging to them are, and the lower the rank of the corresponding label matrix. Due to the introduction of noise, the original correlation of the labels is destroyed. Based on this assumption, applying the low-rank constraint to the ideal label matrix can effectively mine the correlation between labels and samples, weaken the negative impact of label noise, and enhance the ability of each modality to identify samples of the same type. Therefore, the following objective function can be constructed:

[0092]

[0093] in, W is the matrix that maps the labeled samples of each modality to the ideal label matrix: The projection matrix, α and β are balance factors, is a low-rank learning loss function for label noise. The objective function shown in formula (14) is an NP-hard problem. Existing work shows that the nuclear norm is an approximate convex envelope of the rank function, and the nuclear norm can be used to approximate the rank of the matrix. Therefore, formula (14) can be transformed into:

[0094]

[0095] Among them, ||·|| * Represents the nuclear norm of the matrix.

[0096] (5) Construct the overall objective function and learn the projection matrices W1 and W2 for the image modality and text modality respectively;

[0097] Specifically, we consider both intra-modal and inter-modal contrastive learning losses. Structured semantic similarity preserving loss for all samples Cross-modal local relation preserving loss and low-rank learning loss for label noise The total objective function shown in formula (16) is constructed, and the objective function solution method of alternating solution and iterative update is used to learn the projection matrices W1 and W2 for the image modality and text modality respectively. Specifically, the total objective function shown in formula (16) is constructed:

[0098]

[0099] Wherein, W1 and W2 are projection matrices of the image mode and the text mode, respectively. In view of the non-convex nature of the total objective function (16), the embodiment of the present invention uses an alternating solution and iterative update objective function solution method to obtain the solution of the total objective function (16): W, W1 and W2, that is, alternately solve the following four sub-problems: fix W, W1 and W2, solve fixed W1 and W2, solve for W; fixed W and W2, solve for W1; fixed W and W1, solve for W2.

[0100] (6) Generate binary hash codes for query samples and samples in the retrieval sample set based on the learned projection matrices W1 and W2 of the image modality and text modality;

[0101] Specifically, assuming that the feature vector of a query sample in the image modality is The feature vector of a query sample in text mode is The characteristics of samples in the image modality retrieval sample set are: The characteristics of samples in the text modality retrieval sample set are: in, Represents the number of samples in the retrieval sample set; the binary hash codes of the image modality and text modality query samples and samples in the retrieval sample set are: and in, sgn(·) is the sign function.

[0102] (7) Calculate the Hamming distance between the query sample and each sample in the retrieval sample set based on the generated binary hash code;

[0103] Specifically, use the formula Calculate the query sample of the image modality to the first retrieval sample set of the text modality The Hamming distance of samples; using the formula Calculate the query sample of the text modality to the image modality retrieval sample set The Hamming distance of samples.

[0104] (8) Complete cross-modal retrieval of query samples.

[0105] Specifically, the calculated Hamming distance Sort them in ascending order, then take the samples corresponding to the first Z minimum distances in the text modality retrieval sample set as the retrieval results; calculate the Hamming distance Sort them in ascending order, and then take the samples corresponding to the first Z minimum distances in the image modality retrieval sample set as the retrieval results.

[0106] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0107] like Figure 1 As shown, the semi-supervised low-rank cross-modal hash retrieval method for label noise provided by an embodiment of the present invention includes:

[0108] Constructing inter-modality contrast loss function Among them, f1(x 1j ) is the sample x of the first mode 1j The hash function, f2(x 2j ) is the sample x of the second mode 2j The hash function, f1(x 1k ) is the sample x of the first mode 1k The hash function, f2(x 2k ) is the sample x of the second mode 2k Hash function of

[0109] Construct the sample x of the first mode 1j With the enhanced sample x′ 1j Intra-modal contrastive loss function Among them, f1(x′ 1j ) is the sample x′ of the first mode 1j The hash function, f1(x′ 1k ) is the sample x′ of the first mode 1k Hash function of

[0110] Construct the sample x of the second mode 2j With the enhanced sample x′ 2j Intra-modal contrastive loss function Among them, f2(x′ 2j ) is the sample x′ of the second mode 2j The hash function, f2(x′ 2k ) is the sample x′ of the second mode 2k Hash function of

[0111] Constructing the total loss function for intra-modal and inter-modal contrastive learning Among them, λ1 and λ2 are hyperparameters;

[0112] By formula and Calculate the samples x of the first mode respectively 1i and x 1j The cosine similarity between and the sample x of the second mode 2i and x 2j The cosine similarity between

[0113] By formula and Respectively and Mapping is performed to obtain the mapped and

[0114] By formula Calculate the fusion similarity between paired samples of the two modalities

[0115] By formula Calculate the re-weighted fusion similarity Among them, NN t (i) represents the set of the t most recent sample pairs before the i-th sample pair;

[0116] By formula Calculate the structural similarity

[0117] By formula Calculate the structural semantic similarity Among them, ζ∈[0,1] is used to balance the fusion similarity. and structural similarity Parameters;

[0118] Constructing a structured semantic similarity loss function Among them, K is the length of the hash code, f1(x 1p ) is the sample x of the first mode 1p The hash function, f2(x 2q ) is the sample x of the second mode 2q The hash function of For the matrix Elements in

[0119] Define the neighbor affinity matrix A used to capture the local information between different modalities. The element A of the matrix A is ij for: in, represents the k nearest neighbors of x, is the i-th labeled sample of the first modality and the jth labeled sample of the second modality The cross-modal similarity between is the mathematical expectation of the distance of all data pairs of the first mode, and the heat kernel parameter is the mathematical expectation of the distance of all data pairs of the second mode.

[0120] Define the inter-class similarity matrix S b , the matrix S b Elements in for: Define the intra-class similarity matrix S w , the matrix S w Elements in for: Among them, n c is the number of samples in the cth category, is the i-th column vector in the label matrix of the labeled samples of the first modality, is a column vector The cth element of is the jth column vector in the label matrix of the labeled samples of the second modality, is a column vector the cth element of ;

[0121] By the formula S = S w -S b Calculate the semantic similarity matrix S, the element S in S ij for:

[0122] Constructing a cross-modal semantic relevance loss function in, is the i-th labeled sample of the first modality The hash code of is the jth labeled sample of the second modality The hash code of

[0123] Constructing a low-rank learning loss function for label noise Among them, ||·|| * represents the nuclear norm of the matrix, L i is the label matrix containing noisy labels, is the ideal label matrix with labeled samples in the ith modality, is the feature matrix of the labeled samples of the i-th modality, and W is the matrix that maps the labeled samples of each modality to the ideal label matrix. The projection matrix, α and β are balance factors;

[0124] Constructing the overall objective function Learn projection matrices W1 and W2 for the first and second modalities respectively;

[0125] This step is explained in detail to construct the overall objective function The projection matrices W1 and W2 are learned for the first and second modalities respectively, including:

[0126] Fix W, W1 and W2 in the total objective function and solve Fixed total objective function W1 and W2, solve for W; fix the total objective function W and W2, solve W1; fix the total objective function W and W1, solve W2; until the preset number of iterations is reached, the final projection matrices W1 and W2 are obtained.

[0127] By formula Calculate the binary hash code of the query sample of the first modality By formula Calculate the binary hash code of the query sample of the second modality By formula Calculate the binary hash code of the jth sample in the retrieval sample set of the first modality By formula Calculate the binary hash code of the jth sample in the retrieval sample set of the second modality in, sgn(·) is the sign function, is the feature vector of the query sample of the first modality, is the feature vector of the query sample of the second modality, is the feature vector of the jth sample in the retrieval sample set of the first modality, is the feature vector of the jth sample in the retrieval sample set of the second modality;

[0128] By formula Calculate the Hamming distance between the query sample of the first modality and the jth sample in the retrieval sample set of the second modality By formula Calculate the Hamming distance between the query sample of the second modality and the jth sample in the retrieval sample set of the first modality

[0129] Take the first Z minimum Hamming distances in the retrieval sample set of the second modality The corresponding samples are taken as the retrieval results, and the first Z minimum Hamming distances are taken from the retrieval sample set of the first modality The corresponding samples are taken as retrieval results.

[0130] Accordingly, the semi-supervised low-rank cross-modal hash retrieval system for label noise provided by the embodiment of the present invention includes:

[0131] Inter-modal contrast loss function construction module, used to construct inter-modal contrast loss function Among them, f1(x 1j ) is the sample x of the first mode 1j The hash function, f2(x 2j ) is the sample x of the second mode 2j The hash function, f1(x 1k ) is the sample x of the first mode 1k The hash function, f2(x 2k ) is the sample x of the second mode 2k Hash function of

[0132] The contrast loss function construction module in the first modality is used to construct the sample x of the first modality 1j With the enhanced sample x′ 1j Intra-modal contrastive loss function Among them, f1(x′ 1j ) is the sample x′ of the first mode 1j The hash function, f1(x′ 1k ) is the sample x′ of the first mode 1k Hash function of

[0133] The contrast loss function building module in the second modality is used to construct the sample x of the second modality 2j With the enhanced sample x′ 2j Intra-modal contrastive loss function Among them, f2(x′ 2j ) is the sample x′ of the second mode 2j The hash function, f2(x′ 2k ) is the sample x′ of the second mode 2k Hash function of

[0134] The module for building the total loss function of intra-modal and inter-modal contrastive learning is used to build the total loss function of intra-modal and inter-modal contrastive learning. Among them, λ1 and λ2 are hyperparameters;

[0135] The cosine similarity calculation module is used to calculate the cosine similarity through the formula and Calculate the samples x of the first mode respectively 1i and x 1j The cosine similarity between and the sample x of the second mode 2i and x 2j The cosine similarity between

[0136] Mapping module, used to pass formula and Respectively and Mapping is performed to obtain the mapped and

[0137] Fusion similarity calculation module, used to calculate the formula Calculate the fusion similarity between paired samples of the two modalities

[0138] The re-weighted fusion similarity calculation module is used to calculate the similarity through the formula Calculate the re-weighted fusion similarity Among them, NN t (i) represents the set of the t most recent sample pairs before the i-th sample pair;

[0139] The structural similarity calculation module is used to calculate the similarity of Calculate the structural similarity

[0140] The structured semantic similarity calculation module is used to calculate the structured semantic similarity through the formula Calculate the structural semantic similarity Among them, ζ∈[0,1] is used to balance the fusion similarity and structural similarity Parameters;

[0141] Structured semantic similarity loss function building module, used to build structured semantic similarity loss function Among them, K is the length of the hash code, f1(x 1p ) is the sample x of the first mode 1p The hash function, f2(x 2q ) is the sample x of the second mode 2q The hash function of For the matrix Elements in

[0142] The neighbor affinity matrix building module is used to define the neighbor affinity matrix A for capturing the local information between different modalities. The element A of the matrix A ij for: in, represents the k nearest neighbors of x, is the i-th labeled sample of the first modality and the jth labeled sample of the second modality The cross-modal similarity between is the mathematical expectation of the distance of all data pairs of the first mode, and the heat kernel parameter is the mathematical expectation of the distance of all data pairs of the second mode.

[0143] Similarity matrix building module, used to define the inter-class similarity matrix S b , the matrix S b Elements in for: Define the intra-class similarity matrix S w , the matrix S w Elements in for: Among them, n c is the number of samples in the cth category, is the i-th column vector in the label matrix of the labeled samples of the first modality, is a column vector The cth element of is the jth column vector in the label matrix of the labeled samples of the second modality, is a column vector the cth element of ;

[0144] The semantic similarity matrix calculation module is used to calculate the semantic similarity matrix through the formula S = S w -S b Calculate the semantic similarity matrix S, the element S in S ij for:

[0145] Cross-modal semantic relevance loss function construction module, used to construct cross-modal semantic relevance loss function in, is the i-th labeled sample of the first modality The hash code of is the jth labeled sample of the second modality The hash code of

[0146] A module for building low-rank learning loss functions for label noise, which is used to build low-rank learning loss functions for label noise Among them, ||·|| * represents the nuclear norm of the matrix, L i is the label matrix containing noisy labels, is the ideal label matrix with labeled samples in the ith modality, is the feature matrix of the labeled samples of the i-th modality, and W is the matrix that maps the labeled samples of each modality to the ideal label matrix. The projection matrix, α and β are balance factors;

[0147] The overall objective function building module is used to build the overall objective function Learn projection matrices W1 and W2 for the first and second modalities respectively;

[0148] Specifically, the overall objective function construction module is specifically used to construct the overall objective function Fix W, W1 and W2 in the total objective function and solve Fixed total objective function W1 and W2, solve for W; fix the total objective function W and W2, solve W1; fix the total objective function W and W1, solve W2; until the preset number of iterations is reached, the final projection matrices W1 and W2 are obtained.

[0149] The hash code calculation module is used to calculate the Calculate the binary hash code of the query sample of the first modality By formula Calculate the binary hash code of the query sample of the second modality By formula Calculate the binary hash code of the jth sample in the retrieval sample set of the first modality By formula Calculate the binary hash code of the jth sample in the retrieval sample set of the second modality in, sgn(·) is the sign function, is the feature vector of the query sample of the first modality, is the feature vector of the query sample of the second modality, is the feature vector of the jth sample in the retrieval sample set of the first modality, is the feature vector of the jth sample in the retrieval sample set of the second modality;

[0150] The Hamming distance calculation module is used to calculate the distance between Calculate the Hamming distance between the query sample of the first modality and the jth sample in the retrieval sample set of the second modality By formula Calculate the Hamming distance between the query sample of the second modality and the jth sample in the retrieval sample set of the first modality

[0151] The retrieval module is used to obtain the first Z minimum Hamming distances in the retrieval sample set of the second modality. The corresponding samples are taken as the retrieval results, and the first Z minimum Hamming distances are taken from the retrieval sample set of the first modality The corresponding samples are taken as retrieval results.

[0152] The beneficial effects of the present invention are described below in conjunction with specific experiments.

[0153] The relevant experiments implemented for the method of the present invention are carried out on the Pascal VOC 2007 dataset. The PascalVOC 2007 dataset contains 9963 images belonging to 20 categories such as birds, bicycles, cars and ships, and each image is marked with a label. In the experiment, the method of the present invention divides the dataset into a training set containing 5011 image-label pairs and a test set containing 4952 image-label pairs. For the training set, 3% of the samples are added with label noise. The image modality uses the original pixel features as input features. The text modality uses the 399-dimensional word frequency features as input features. In this experiment, two cross-modal retrieval tasks are mainly carried out, namely: retrieving text with images and retrieving images with text, which are represented by Img2Txt and Txt2Img respectively.

[0154] The embodiment of the present invention uses the Mean Average Precision (MAP) to evaluate the performance of the cross-modal hash retrieval method. In order to obtain MAP, the average precision (AP) of each query sample is first calculated. After obtaining the average AP of all query samples, the MAP is obtained by averaging all APs. The embodiment of the present invention uses a small batch gradient descent algorithm, and the batch size is set to 128. In this experiment, a 5-fold cross validation is used to determine the parameter β in the method of the present invention. i , γ1 and γ2. For the parameters in other methods, the parameters are set according to the parameter setting principles recommended by each method. The results reported in this experiment are the average of 10 random experimental results.

[0155] The methods compared with the method of the present invention are: Deep Evidential Cross-modal Learning (DECL) method and Noise-robust Deep Cross-Modal Hashing (NrDCMH) method. Table 1 lists the mean average precision (MAP) of the method of the present invention and the comparison method when performing cross-modal hash retrieval on the Pascal VOC 2007 dataset. As can be seen from Table 1, for two retrieval tasks, under three hash code lengths, the cross-modal retrieval performance of the method of the present invention is better than that of the comparison method, which shows that the method of the present invention is an effective cross-modal hash retrieval method.

[0156] Table 1 MAP of each method on Pascal VOC 2007 dataset

[0157]

[0158] In summary, the embodiments of the present invention effectively weaken the adverse effects of label noise and improve the cross-modal retrieval accuracy facing the influence of label noise.

[0159] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0163] The parts not described in detail in the embodiments of the present invention are all known technologies to those skilled in the art. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A semi-supervised low-rank cross-modal hashing retrieval method for label noise, characterized in that: include: Constructing inter-modality contrast loss function Among them, f1(x 1j ) is the sample x of the first mode 1j The hash function, f2(x 2j ) is the sample x of the second mode 2j The hash function, f1(x 1k ) is the sample x of the first mode 1k The hash function, f2(x 2k ) is the sample x of the second mode 2k Hash function of Construct the sample x of the first mode 1j With the enhanced sample x′ 1j Intra-modal contrastive loss function Among them, f1(x′ 1j ) is the sample x′ of the first mode 1j The hash function, f1(x′ 1k ) is the sample x′ of the first mode 1k Hash function of Construct the sample x of the second mode 2j With the enhanced sample x′ 2j Intra-modal contrastive loss function Among them, f2(x′ 2j ) is the sample x′ of the second mode 2j The hash function, f2(x′ 2k ) is the sample x′ of the second mode 2k Hash function of Constructing the total loss function for intra-modal and inter-modal contrastive learning Among them, λ1 and λ2 are hyperparameters; By formula and Calculate the samples x of the first mode respectively 1i and x 1j The cosine similarity between and the sample x of the second mode 2i and x 2j The cosine similarity between By formula and Respectively and Mapping is performed to obtain the mapped and By formula Calculate the fusion similarity between paired samples of the two modalities By formula Calculate the re-weighted fusion similarity Among them, NN t (i) represents the set of the t most recent sample pairs before the i-th sample pair; By formula Calculate the structural similarity By formula Calculate the structural semantic similarity Among them, ζ∈[0,1] is used to balance the fusion similarity and structural similarity Parameters; Constructing a structured semantic similarity loss function Among them, K is the length of the hash code, f1(x 1p ) is the sample x of the first mode 1p The hash function, f2(x 2q ) is the sample x of the second mode 2q The hash function of For the matrix Elements in Define the neighbor affinity matrix A used to capture the local information between different modalities. The element A of the matrix A is ij for: in, represents the k nearest neighbors of x, is the i-th labeled sample of the first modality and the jth labeled sample of the second modality The cross-modal similarity between is the mathematical expectation of the distance of all data pairs of the first mode, and the heat kernel parameter is the mathematical expectation of the distance of all data pairs of the second mode; Define the inter-class similarity matrix S b , the matrix S b Elements in for: Define the intra-class similarity matrix S w , the matrix S w Elements in for: Among them, n c is the number of samples in the cth category, is the i-th column vector in the label matrix of the labeled samples of the first modality, is a column vector The cth element of is the jth column vector in the label matrix of the labeled samples of the second modality, is a column vector the cth element of ; By the formula S = S w -S b Calculate the semantic similarity matrix S, the element S in S ij for: Constructing a cross-modal semantic relevance loss function in, is the i-th labeled sample of the first modality The hash code of is the jth labeled sample of the second modality The hash code of Constructing a low-rank learning loss function for label noise Among them, ||·|| * represents the nuclear norm of the matrix, L i is the label matrix containing noisy labels, is the ideal label matrix with labeled samples in the ith modality, is the feature matrix of the labeled samples of the i-th modality, and W is the matrix that maps the labeled samples of each modality to the ideal label matrix. The projection matrix, α and β are balance factors; Constructing the overall objective function Learn projection matrices W1 and W2 for the first and second modalities respectively; By formula Calculate the binary hash code of the query sample of the first modality By formula Calculate the binary hash code of the query sample of the second modality By formula Calculate the binary hash code of the jth sample in the retrieval sample set of the first modality By formula Calculate the binary hash code of the jth sample in the retrieval sample set of the second modality in, sgn(·) is the sign function, is the feature vector of the query sample of the first modality, is the feature vector of the query sample of the second modality, is the feature vector of the jth sample in the retrieval sample set of the first modality, is the feature vector of the jth sample in the retrieval sample set of the second modality; By formula Calculate the Hamming distance between the query sample of the first modality and the jth sample in the retrieval sample set of the second modality By formula Calculate the Hamming distance between the query sample of the second modality and the jth sample in the retrieval sample set of the first modality Take the first Z minimum Hamming distances in the retrieval sample set of the second modality The corresponding samples are taken as the retrieval results, and the first Z minimum Hamming distances are taken from the retrieval sample set of the first modality The corresponding samples are taken as retrieval results.

2. The semi-supervised low-rank cross-modal hash retrieval method for label noise as claimed in claim 1, characterized in that: The overall objective function is constructed The projection matrices W1 and W2 are learned for the first and second modalities respectively, including: Fix W, W1 and W2 in the overall objective function and solve Fix the total objective function W1 and W2, solve W; fix the total objective function W and W2, solve W1; fix the total objective function W and W1, solve W2; until the preset number of iterations is reached, the final projection matrices W1 and W2 are obtained.

3. A semi-supervised low-rank cross-modal hash retrieval system for label noise, characterized in that: include: Inter-modal contrast loss function construction module, used to construct inter-modal contrast loss function Among them, f1(x 1j ) is the sample x of the first mode 1j The hash function, f2(x 2j ) is the sample x of the second mode 2j The hash function, f1(x 1k ) is the sample x of the first mode 1k The hash function, f2(x 2k ) is the sample x of the second mode 2k Hash function of The contrast loss function construction module in the first modality is used to construct the sample x of the first modality 1j With the enhanced sample x′ 1j Intra-modal contrastive loss function Among them, f1(x′ 1j ) is the sample x′ of the first mode 1j The hash function, f1(x′ 1k ) is the sample x′ of the first mode 1k Hash function of The contrast loss function building module in the second modality is used to construct the sample x of the second modality 2j With the enhanced sample x′ 2j Intra-modal contrastive loss function Among them, f2(x′ 2j ) is the sample x′ of the second mode 2j The hash function, f2(x′ 2k ) is the sample x′ of the second mode 2k Hash function of The module for building the total loss function of intra-modal and inter-modal contrastive learning is used to build the total loss function of intra-modal and inter-modal contrastive learning. Among them, λ1 and λ2 are hyperparameters; The cosine similarity calculation module is used to calculate the cosine similarity through the formula and Calculate the samples x of the first mode respectively 1i and x 1j The cosine similarity between and the sample x of the second mode 2i and x 2j The cosine similarity between Mapping module, used to pass formula and Respectively and Mapping is performed to obtain the mapped and Fusion similarity calculation module, used to calculate the formula Calculate the fusion similarity between paired samples of the two modalities The re-weighted fusion similarity calculation module is used to calculate the similarity through the formula Calculate the re-weighted fusion similarity Among them, NN t (i) represents the set of the t most recent sample pairs before the i-th sample pair; The structural similarity calculation module is used to calculate the similarity of Calculate the structural similarity The structured semantic similarity calculation module is used to calculate the structured semantic similarity through the formula Calculate the structural semantic similarity Among them, ζ∈[0,1] is used to balance the fusion similarity and structural similarity Parameters; Structured semantic similarity loss function building module, used to build structured semantic similarity loss function Among them, K is the length of the hash code, f1(x 1p ) is the sample x of the first mode 1p The hash function, f2(x 2q ) is the sample x of the second mode 2q The hash function of For the matrix Elements in The neighbor affinity matrix building module is used to define the neighbor affinity matrix A for capturing the local information between different modalities. The element A of the matrix A ij for: in, represents the k nearest neighbors of x, is the i-th labeled sample of the first modality and the jth labeled sample of the second modality The cross-modal similarity between is the mathematical expectation of the distance of all data pairs of the first mode, and the heat kernel parameter is the mathematical expectation of the distance of all data pairs of the second mode; Similarity matrix building module, used to define the inter-class similarity matrix S b , the matrix S b Elements in for: Define the intra-class similarity matrix S w , the matrix S w Elements in for: Among them, n c is the number of samples in the cth category, is the i-th column vector in the label matrix of the labeled samples of the first modality, is a column vector The cth element of is the jth column vector in the label matrix of the labeled samples of the second modality, is a column vector the cth element of ; The semantic similarity matrix calculation module is used to calculate the semantic similarity matrix through the formula S = S w -S b Calculate the semantic similarity matrix S, the element S in S ij for: Cross-modal semantic relevance loss function construction module, used to construct cross-modal semantic relevance loss function in, is the i-th labeled sample of the first modality The hash code of is the jth labeled sample of the second modality The hash code of A module for building low-rank learning loss functions for label noise, which is used to build low-rank learning loss functions for label noise Among them, ||·|| * represents the nuclear norm of the matrix, L i is the label matrix containing noisy labels, is the ideal label matrix with labeled samples in the ith modality, is the feature matrix of the labeled samples of the i-th modality, and W is the matrix that maps the labeled samples of each modality to the ideal label matrix. The projection matrix, α and β are balance factors; The overall objective function building module is used to build the overall objective function Learn projection matrices W1 and W2 for the first and second modalities respectively; The hash code calculation module is used to calculate the Calculate the binary hash code of the query sample of the first modality By formula Calculate the binary hash code of the query sample of the second modality By formula Calculate the binary hash code of the jth sample in the retrieval sample set of the first modality By formula Calculate the binary hash code of the jth sample in the retrieval sample set of the second modality in, sgn(·) is the sign function, is the feature vector of the query sample of the first modality, is the feature vector of the query sample of the second modality, is the feature vector of the jth sample in the retrieval sample set of the first modality, is the feature vector of the jth sample in the retrieval sample set of the second modality; The Hamming distance calculation module is used to calculate the distance between Calculate the Hamming distance between the query sample of the first modality and the jth sample in the retrieval sample set of the second modality By formula Calculate the Hamming distance between the query sample of the second modality and the jth sample in the retrieval sample set of the first modality The retrieval module is used to obtain the first Z minimum Hamming distances in the retrieval sample set of the second modality. The corresponding samples are taken as the retrieval results, and the first Z minimum Hamming distances are taken from the retrieval sample set of the first modality The corresponding samples are taken as retrieval results.

4. The semi-supervised low-rank cross-modal hash retrieval system for label noise as claimed in claim 3, characterized in that: The overall objective function construction module is specifically used to construct the overall objective function Fix W, W1 and W2 in the overall objective function and solve Fix the total objective function W1 and W2, solve W; fix the total objective function W and W2, solve W1; fix the total objective function W and W1, solve W2; until the preset number of iterations is reached, the final projection matrices W1 and W2 are obtained.

Images