A cross-modal hash retrieval method based on boundary mutual information

Abstract

A cross-modal hash retrieval method based on boundary mutual information includes the following steps: step 1, building a network, constructing a feature learning network and a hash learning network, wherein the feature learning network and the hash learning network of each modality are both composed of a generative adversarial network composed of multiple layers of fully connected layers; step 2, constructing a hash function; step 3, constructing an objective function composed of a cross entropy loss, a sorted cosine ternary loss, a boundary mutual information loss, and a discrete loss with an orthogonal constraint; step 4, optimizing the objective function; step 5, dividing a data set into a query set Q and a to-be-queried set P, and taking a part of the to-be-queried set P as a training set T, taking the training set data and label information as network input, and initializing network parameters and hash codes, repeating step 4, iterating to obtain the optimal network parameters and hash codes, and obtaining the optimal hash function according to step 2; step 6, cross-modal retrieval and precision testing. The present invention optimizes convergence and has high retrieval precision.

Description

Technical Field

[0001] The present invention relates to big data processing and analysis in the field of computer vision, and in particular to a hash optimization algorithm and a cross-modal retrieval method. Background Art

[0002] With the continuous development and breakthrough of technology, information begins to be transmitted through multimedia, and large-scale cross-modal retrieval has been widely used in daily life. Cross-modal retrieval is an information retrieval task that aims to retrieve relevant content or data from different media or modalities. Due to the different feature representations of different modalities, the semantic gap is still a problem that needs to be solved urgently. At the same time, the time cost and storage cost of the retrieval task are rising rapidly. As an efficient method, hashing has been widely adopted due to its advantages of small storage capacity and high retrieval efficiency, and cross-modal hash retrieval has emerged. Cross-modal hash retrieval is a cross-modal retrieval task that maps data from different modalities into a shared hash space by constructing a hash function so that relevant content can be retrieved quickly and efficiently between different modalities. On the one hand, the hash function maps high-dimensional original features into low-dimensional hash codes while retaining the original similar structure, which greatly reduces the computational cost and improves the retrieval speed. On the other hand, designing effective loss functions and matching strategies is the key to cross-modal hash retrieval. These loss functions usually measure the similarity between modalities in the hash space so that relevant cross-modal results can be obtained during retrieval. However, how to bridge the heterogeneous gap, maintain semantic consistency, and reduce information redundancy are still problems that need to be solved in cross-modal hash retrieval. At the same time, because hash codes are discrete, their optimization problem is an NP problem. How to optimize them efficiently has become another difficult problem in this field. Summary of the invention

[0003] In order to overcome the problems of heterogeneous gap and information redundancy existing in the existing cross-modal hash retrieval methods, the present invention provides a cross-modal hash retrieval method based on boundary mutual information with convergence of optimization algorithm and high retrieval accuracy, introduces adversarial learning to reduce the heterogeneous gap, and proposes an objective function based on boundary mutual information to maintain semantic consistency, and adds orthogonal constraints to the hash code to reduce information redundancy, and optimizes the hash function by using the alternating direction multiplier method (ADMM) algorithm and the stochastic gradient descent method to obtain a hash function, uses the hash function to convert images and texts into hash codes, and uses the hash codes for image and text retrieval during retrieval.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] A cross-modal hash retrieval method based on boundary mutual information, characterized in that the method comprises the following steps:

[0006] Step 1: Construct a feature learning network and a hash learning network. Both the feature learning network and the hash learning network are composed of a generative adversarial network consisting of multiple layers of fully connected layers. The process is as follows:

[0007] Step 1.1: Construct a feature learning network, which consists of two generating networks and one discriminative network;

[0008] Step 1.2: Construct a hash learning network. The hash learning network consists of two generation networks and one discrimination network.

[0009] Step 2: Construct the hash function and quantize the loss. The process is as follows:

[0010] Step 2.1: Perform symbolic operations on the final output of the hash learning network;

[0011] Step 2.2: The definition of hash function is the mapping function from feature space to Hamming space. Construct the hash function from the definition of hash function;

[0012] Step 2.3: Construct quantization loss. The quantization loss consists of discrete loss and adds orthogonal constraints to reduce the information redundancy of hash codes;

[0013] Step 3: Similarity preservation loss is constructed, which consists of sorted cosine ternary loss and boundary mutual information loss;

[0014] Step 4: Objective function optimization, using the alternating optimization method to update the network parameters and hash code, the process is as follows:

[0015] Step 4.1: Fix the hash code and use the back propagation algorithm to optimize the network parameters;

[0016] Step 4.2: Fix the network parameters and optimize the hash code using the alternating direction multiplication method;

[0017] Step 5: Network training and hash function learning, the process is as follows:

[0018] Step 5.1: Divide the data set into the query set Q and the query set P, and take a part of the query set P as the training set T;

[0019] Step 5.2: Use the VGG network to extract the deep features of the image and use the bag-of-words model to obtain the representation vector of the text;

[0020] Step 5.3: Input the image depth features, text representation vector and label information of the training set;

[0021] Step 5.4: Randomly initialize network parameters;

[0022] Step 5.5: According to step 4, iterate and optimize from step 4.1 to 4.2 until convergence;

[0023] Step 5.6: According to the hash function constructed in step 2.2, the optimal network parameters are brought in to obtain the hash function;

[0024] Step 6: Image and text retrieval and accuracy testing.

[0025] Further, the process of step 6 is as follows:

[0026] Step 6.1: Input the query set Q and the query set P into the trained feature learning network and hash function to obtain the hash codes Q+, Q- of the image text in the query set Q and the hash codes P+, P- of the image text in the query set P respectively;

[0027] Step 6.2: Image retrieval text, calculate the Hamming distance between the hash code Q+ of each image in the query set Q and the hash code P- of all texts in the query set P, and sort them from small to large according to the Hamming distance as the retrieval results;

[0028] Step 6.3: Text retrieval image, calculate the Hamming distance between the hash code Q- of each text in the query set Q and the hash code P+ of all images in the query set P, and sort them from small to large according to the Hamming distance as the retrieval results;

[0029] Step 6.4: Compare the query image or text label with the label corresponding to the retrieved text or image, calculate the average accuracy of all query images or texts according to the evaluation criteria in information retrieval, and output the calculation result.

[0030] Further, the process of step 1.1 is as follows:

[0031] Step 1.1.1: Construct a discriminative network consisting of three fully connected layers, with ReLU as the activation function of each layer;

[0032] Step 1.1.2: In the image modality, a generative network consisting of three fully connected layers is constructed, and ReLU is used as the activation function of each layer;

[0033] Step 1.1.3: In the text mode, a generative network consisting of three fully connected layers is constructed, and ReLU is used as the activation function of each layer;

[0034] Step 1.1.4: Feature learning part adversarial learning, construct the adversarial loss of the feature learning network, that is, the cross entropy loss.

[0035] Furthermore, the process of step 1.2 is as follows:

[0036] Step 1.2.1: Construct a discriminant network with two fully connected layers, and use ReLU as the activation function of each layer;

[0037] Step 1.2.2: On the image modality, construct a single-layer fully connected generative network with the tanh function as the activation function;

[0038] Step 1.2.3: In the text mode, construct a single-layer fully connected generative network with the tanh function as the activation function;

[0039] Step 1.2.4: Hash learning part adversarial learning, construct the adversarial loss of the hash learning network, that is, the cross entropy loss.

[0040] Beneficial effects of the present invention: On the one hand, in order to align the features of different modalities, adversarial learning is introduced to learn better features. On the other hand, in order to maintain the original semantic correlation between different modalities in the process of generating hash codes, boundary mutual information is proposed to minimize the overlap of Hamming distribution, which can reduce boundary ambiguity and maximize the distance between samples of different categories in the Hamming space. In order to reduce information redundancy and generate higher quality hash codes, orthogonal constraints are added to the discrete loss. The addition of adversarial learning, boundary mutual information and orthogonal constraints effectively improves the retrieval accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of the framework of the cross-modal hash retrieval method based on boundary mutual information. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0043] Reference Figure 1 , a cross-modal hash retrieval method based on boundary mutual information, including six processes: network construction, hash function construction, objective function construction, objective function optimization, network training and hash function learning, image and text retrieval and accuracy testing.

[0044] The cross-modal hash retrieval method based on boundary mutual information comprises the following steps:

[0045] Step 1: Construct a feature learning network and a hash learning network. Both the feature learning network and the hash learning network are composed of a generative adversarial network consisting of multiple layers of fully connected layers. The process is as follows:

[0046] Step 1.1: Construct a feature learning network. The feature learning network consists of two generation networks and one discriminant network. The process is as follows:

[0047] Step 1.1.1: Construct a discriminant network DF consisting of three fully connected layers P , ReLU is used as the activation function for each layer;

[0048] Step 1.1.2: On the image modality, construct a generative network GF consisting of three fully connected layers I , ReLU is used as the activation function of each layer, and then the image features are obtained:

[0049]

[0050] in, represents the parameters of the image modality generation network, Represents the deep features of the image extracted by VGG-F;

[0051] Step 1.1.3: In the text mode, a generative network consisting of three fully connected layers is constructed, and ReLU is used as the activation function of each layer to obtain text features:

[0052]

[0053] in, represents the parameters of the image modality generation network, Represents the representation vector obtained by the bag-of-words vector of the text;

[0054] Step 1.1.4: Generate network GF in image mode I The output F of is taken as the true feature, and the text modality generation network GF T The output V is used as a false feature and the discriminant network DF is used P Perform identification and then construct feature adversarial loss to align features of different modalities and reduce the heterogeneous gap:

[0055]

[0056] Step 1.2: Construct a hash learning network. The hash learning network consists of two generation networks and one discrimination network. The process is as follows:

[0057] Step 1.2.1: Construct a discriminant network DH consisting of two fully connected layers P , ReLU is used as the activation function for each layer;

[0058] Step 1.2.2: On the image modality, construct a generative network GH consisting of a single fully connected layer I , tanh function is used as the activation function, and then the pseudo hash code of the image modality is obtained:

[0059]

[0060] in, Represents the parameters of the image modality generation network.

[0061] Step 1.2.3: In the text mode, a generative network consisting of a single fully connected layer is constructed, and the tanh function is used as the activation function;

[0062]

[0063] in, Represents the parameters of the text modality generation network.

[0064] Step 1.2.4: Generate network GH in image modality I Output H I As a true feature, the text modality generation network GH T Output H T As a false feature, use the discriminant network DH P To identify, we can construct a hash adversarial loss to align the pseudo-hash codes of different modalities and reduce the heterogeneous gap:

[0065]

[0066] Step 2: Construct a hash function. The process is as follows:

[0067] Step 2.1: Perform symbolic operations on the final output of the hash learning network to obtain the hash codes B of the image modality and text modality I and B T :

[0068] B I =sgn(H I ) (7)

[0069] B T =sgn(H T ) (8)

[0070] in, represents a symbolic function;

[0071] Step 2.2: Build a hash function;

[0072]

[0073]

[0074] Step 3: Objective function construction, which consists of quantization loss, similarity preservation loss, and adversarial loss. The process is as follows:

[0075] Step 3.1: To generate high-quality hash code B, construct a quantization loss, i.e., a discrete loss with orthogonal constraints, to reduce the quantization error and information redundancy of the hash code:

[0076]

[0077] Among them, I N Represents the N-order identity matrix, and N represents the length of the hash code.

[0078] Step 3.2: Construct a similarity preservation loss, which consists of boundary mutual information loss and sorting ternary loss; the process is as follows:

[0079] Step 3.2.1: Construct boundary mutual information loss, which aims to maintain the consistency of the overall distribution in the hash mapping process while maintaining semantic similarity; since the Hamming distribution reflects the spatial distribution of samples in the Hamming space, it is hoped that the overall difference in the Hamming distribution between similar and dissimilar samples should be as large as possible, which is consistent with reducing the overlap of the two distributions. In the Hamming space, the overlap is reflected in the lack of separation between partially similar and dissimilar samples; the purpose of minimizing the overlap is to make the boundaries between samples of different classes clearer, that is, to effectively separate samples of different classes. For this purpose, the boundary mutual information loss is constructed. The specific construction process is as follows:

[0080] First, we consider the single point case. Without loss of generality, we take the single point of the image modality as the query point;

[0081] Step 3.2.1.1: Calculate the Hamming distance using the following formula:

[0082]

[0083] Step 3.2.1.2: Define two random variables and The features of the sample and the query point are mapped from the feature space to the Hamming space to obtain the Hamming distance between the two. Indicates whether the sample is similar to the query point;

[0084] Then, from the definition of mutual information I(X;Y)=E(X)-E(X|Y), we can get and The mutual information between:

[0085]

[0086] Where E(x) = -∑p i (x)log(p i (x)) represents information entropy;

[0087] To calculate formula (13), we need to first obtain three probability distributions: represents the conditional probability distribution of samples that are not similar to the query point, represents the conditional probability distribution of samples similar to the query point, Represents the probability distribution of the population; by maximizing mutual information, we can reduce and The overlap between them can better separate samples of different categories;

[0088] Step 3.2.1.3: Since the method is supervised, similarity relations are obtained by labeling L = [l1,l2,…,l n ], two samples are similar if they share at least one label, so the similarity matrix is ​​defined as:

[0089]

[0090] Step 3.2.1.4: Since the above three probability distributions are difficult to obtain directly, they are estimated using frequencies. At the same time, the similarity matrix represents the similarity relationship, and then the estimates of the above three probability distributions are obtained:

[0091]

[0092]

[0093]

[0094] Among them, 1[X] represents the indicator function, indicating which elements belong to X;

[0095] Step 3.2.1.5: Considering that the indicator function is not differentiable, use Approximation to (15)(16)(17):

[0096]

[0097]

[0098]

[0099] in,

[0100] Step 3.2.1.6: Since the hash code is discrete, the calculation formula (12) of the Hamming distance is not differentiable. Considering the discrete constraint, the pseudo-hash code is used instead, and (12) is transformed into the following:

[0101]

[0102] Step 3.2.1.7: Combining (18)(19)(20), we get three probability distributions:

[0103]

[0104]

[0105]

[0106] Step 3.2.1.8: According to (13), use (22)(23)(24) to get the mutual information in the single point case:

[0107]

[0108]

[0109] Step 3.2.1.9: Finally, expand the single point situation to all data points:

[0110]

[0111]

[0112] Step 3.2.1.10: The total boundary mutual information loss is obtained as follows:

[0113]

[0114] Step 3.2.2: Construct a sorting ternary loss, that is, to maintain semantic consistency under multiple labels:

[0115]

[0116]

[0117] L total-tri =L I-T +L T-I (31)

[0118] Among them, ω jk Represents the weight factor, which is determined by the normalized discounted cumulative gain NDCG:

[0119]

[0120] Among them, r i is the i-th data point in the ranking list, calculated based on the number of common labels shared by query q from one modality and query i from another modality, and Z is the normalized index, which is obtained as follows:

[0121]

[0122] Step 3.3: Construct the adversarial loss, which is composed of feature adversarial loss and hash adversarial loss. The adversarial loss is obtained through steps 1.1.4 and 1.2.4.

[0123] Step 3.4: Objective function construction. From step 3.1, step 3.2, and step 3.3, we get the overall objective function:

[0124]

[0125] Step 4: Optimize the objective function, using an alternating update optimization strategy. The process is as follows:

[0126] Step 4.1: Fix B, Using the back propagation algorithm renew;

[0127] Step 4.2: Fix B, Using the back propagation algorithm renew;

[0128] Step 4.3: Fix B, Using the back propagation algorithm renew;

[0129] Step 4.4: Fixing Use the alternating direction method of multipliers (ADMM) to update B. The process is as follows:

[0130] Step 4.4.1: Combine (11) to obtain the augmented Lagrangian form:

[0131]

[0132] Where ρ represents the penalty factor;

[0133] Step 4.4.2: Combine (35) to obtain the ADMM iteration form:

[0134]

[0135]

[0136] Λ k =Λ k-1 +ρ(B k -C k ) (36)

[0137] Step 4.4.3: Fix Λ,B, and update C:

[0138] Rewrite the first equation of (36) as:

[0139]

[0140] Since C={-1,1} n×N , C can be calculated as:

[0141] C=sgn(ρB+Λ) (38)

[0142] Step 4.4.4: Fix Λ,C, and update B:

[0143] Similarly, the second equation in (36) can be rewritten as:

[0144]

[0145] Further rewritten as:

[0146]

[0147] Where M = 2H I +2H T +ρC-Λ.

[0148] M is obtained by singular value decomposition:

[0149]

[0150] Among them, U=[u i ] and V = [v i ] represents a triangular matrix, Δ represents a diagonal matrix composed of singular values, and r represents the rank of M;

[0151] If M is full rank, combined with the orthogonal constraint, B is decomposed by SVD to obtain:

[0152]

[0153] Among them, U and V represent triangular matrices;

[0154] If M is not full rank, for the matrix IU T Perform QR decomposition on U and take the first nr columns of Q as Obviously, Similarly, Then we get B:

[0155]

[0156] Combining (42) and (43), we get the optimal iterative solution of B:

[0157]

[0158] Step 4.4.5: Fix B, C, and update Λ:

[0159] Λ k =Λ k-1 +ρ(B k -C k ). (45)

[0160] Step 4.4.5: Combine (36)(38)(44)(45) to obtain the ADMM iterative form:

[0161] Ck =sgn(ρB k-1 +Λ k-1 )

[0162]

[0163] Λ k =Λ k-1 +ρ(B k -C k ) (46)

[0164] Step 5: Network training and hash function learning, the process is as follows:

[0165] Step 5.1: Divide the data set into the query set Q and the query set P, and take a part of the query set P as the training set T;

[0166] Step 5.2: Use the VGG network to extract the deep features of the image and use the bag-of-words model to obtain the representation vector of the text;

[0167] Step 5.3: Input the image depth features, text representation vector and label information of the training set;

[0168] Step 5.4: Randomly initialize network parameters and hash codes;

[0169] Step 5.5: According to step 4, iterate and optimize from step 4.1 to 4.4 until convergence;

[0170] Step 5.6: Substitute the optimal network parameters into (9)(10) to obtain the hash function;

[0171] Step 6: Image and text retrieval and accuracy test. The process is as follows:

[0172] Step 6.1: Input the query set Q and the query set P into the trained feature learning network and hash function to obtain the hash code Q of the image text in the query set Q respectively. + ,Q - , the hash code P of the image text in the query set P + , P - ;

[0173] Step 6.2: Image retrieval text, calculate the hash code Q of each image in the query set Q + and the hash code P of all the texts in the query set P - The Hamming distance between them is sorted from small to large as the search results;

[0174] Step 6.3: Text retrieval image, calculate the hash code Q of each text in the query set Q - and the hash code P of all images in the query set P +The Hamming distance between them is sorted from small to large as the search results;

[0175] Step 6.4: Compare the query image or text label with the label corresponding to the retrieved text or image, calculate the average accuracy of all query images or texts according to the evaluation criteria in information retrieval, and output the calculation result;

[0176] Finally, the retrieval task can be completed through the above steps.

[0177] The specific description above further illustrates in detail the purpose, technical solutions and beneficial effects of the invention. It should be understood that the above is only a specific embodiment of the present invention, which is used to explain the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cross-modal hash retrieval method based on boundary mutual information, characterized in that: The method comprises the following steps: Step 1: Construct a feature learning network and a hash learning network. Both the feature learning network and the hash learning network are composed of a generative adversarial network consisting of multiple layers of fully connected layers. The process is as follows: Step 1.1: Construct a feature learning network, which consists of two generating networks and one discriminative network; Step 1.2: Construct a hash learning network. The hash learning network consists of two generation networks and one discrimination network. Step 2: Construct the hash function and quantize the loss. The process is as follows: Step 2.1: Perform symbolic operations on the final output of the hash learning network; Step 2.2: The definition of the hash function is a mapping function from the feature space to the Hamming space. The hash function is constructed based on the definition of the hash function. Step 2.3: Construct a quantization loss, which consists of a discrete loss and adds an orthogonal constraint to reduce the information redundancy of the hash code; Step 3: Similarity preservation loss is constructed, which consists of sorted cosine ternary loss and boundary mutual information loss; Step 4: Objective function optimization, using the alternating optimization method to update the network parameters and hash code, the process is as follows: Step 4.1: Fix the hash code and use the back propagation algorithm to optimize the network parameters; Step 4.2: Fix the network parameters and optimize the hash code using the alternating direction multiplication method; Step 5: Network training and hash function learning, the process is as follows: Step 5.1: Divide the data set into the query set Q and the query set P, and take a part of the query set P as the training set T; Step 5.2: Use the VGG network to extract the deep features of the image and use the bag-of-words model to obtain the representation vector of the text; Step 5.3: Input the image depth features, text representation vector and label information of the training set; Step 5.4: Randomly initialize network parameters; Step 5.5: According to step 4, iterate and optimize from step 4.1 to 4.2 until convergence; Step 5.6: According to the hash function constructed in step 2.2, the optimal network parameters are brought in to obtain the hash function; Step 6: Image and text retrieval and accuracy testing.

2. A cross-modal hash retrieval method based on boundary mutual information as claimed in claim 1, characterized in that: The process of step 6 is as follows: Step 6.1: Input the query set Q and the query set P into the trained feature learning network and hash function to obtain the hash codes Q+, Q- of the image text in the query set Q and the hash codes P+, P- of the image text in the query set P respectively; Step 6.2: Image retrieval text, calculate the Hamming distance between the hash code Q+ of each image in the query set Q and the hash code P- of all texts in the query set P, and sort them from small to large according to the Hamming distance as the retrieval results; Step 6.3: Text retrieval image, calculate the Hamming distance between the hash code Q- of each text in the query set Q and the hash code P+ of all images in the query set P, and sort them from small to large according to the Hamming distance as the retrieval results; Step 6.4: Compare the query image or text label with the label corresponding to the retrieved text or image, calculate the average accuracy of all query images or texts according to the evaluation criteria in information retrieval, and output the calculation result.

3. A cross-modal hash retrieval method based on boundary mutual information as described in claim 1 or 2, characterized in that: The process of step 1.1 is as follows: Step 1.1.1: Construct a discriminative network consisting of three fully connected layers, with ReLU as the activation function of each layer; Step 1.1.2: In the image modality, a generative network consisting of three fully connected layers is constructed, and ReLU is used as the activation function of each layer; Step 1.1.3: In the text mode, a generative network consisting of three fully connected layers is constructed, and ReLU is used as the activation function of each layer; Step 1.1.4: Feature learning part adversarial learning, construct the adversarial loss of the feature learning network, that is, the cross entropy loss.

4. A cross-modal hash retrieval method based on boundary mutual information as described in claim 1 or 2, characterized in that: The process of step 1.2 is as follows: Step 1.2.1: Construct a discriminant network with two fully connected layers, and use ReLU as the activation function of each layer; Step 1.2.2: On the image modality, construct a single-layer fully connected generative network with the tanh function as the activation function; Step 1.2.3: In the text mode, construct a single-layer fully connected generative network with the tanh function as the activation function; Step 1.2.4: Hash learning part adversarial learning, construct the adversarial loss of the hash learning network, that is, the cross entropy loss.

Images