A multimodal hash retrieval method guided by Hadamard matrix

Through the multimodal hashing method guided by Hadamard matrix, combined with subspace learning and matrix decomposition, the target hash code is generated, which solves the problems of information redundancy and complex hyperparameter optimization in multimodal retrieval, and achieves more efficient multi- Modal retrieval performance and feature representation capability.

CN114637751BActive Publication Date: 2024-11-26ZHENGZHOU UNIVERSITY OF LIGHT INDUSTRY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210304220.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-25
Publication Date
2024-11-26
Estimated Expiration
2042-03-25

AI Technical Summary

Technical Problem

The existing multimodal hashing methods fail to make full use of high-level semantic information in retrieval performance, and the hyperparameter optimization is complex, resulting in limited promotion in practical applications.

Method used

Using the multimodal hashing method guided by Hadamard matrix, the complementary information between multimodal data is adaptively captured through joint subspace learning and hash learning, and the multimodal space is reconstructed through integrated matrix decomposition, and the Hadamard matrix is ​​used. The column properties of the target hash code generate the target hash code, and soft binary constraints are used to improve the discriminantity of the model.

Benefits of technology

It improves multimodal retrieval performance, enhances feature representation ability, reduces quantization loss, and improves the quality and retrieval accuracy of hash code.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114637751B_ABST
    Figure CN114637751B_ABST
Patent Text Reader

Abstract

The present invention discloses a multimodal hash retrieval method guided by a Hadamard matrix. The method jointly performs subspace learning and hash learning in an overall framework. On the one hand, the proposed method adaptively captures the complementary information between multimodal data. In addition, the multimodal space is reconstructed through an integrated matrix decomposition method to enhance the feature representation capability. On the other hand, the method utilizes the column attributes of the Hadamard matrix to generate the target code for each category, and adopts soft binary constraints on the target binary matrix to improve the discriminability of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of hash retrieval, and in particular relates to a multi-modal hash retrieval method guided by a Hadamard matrix. Background Art

[0002] With the rapid growth of multimedia data in social networks, media information retrieval faces huge challenges. As an effective representation method, hashing has attracted widespread attention in large-scale data applications due to its advantages such as low storage and fast retrieval speed. In the past decade, single-mode hashing retrieval techniques have been widely studied and successfully applied to solve large-scale image retrieval problems. However, single-mode hashing retrieval techniques are mostly limited to retrieval tasks within a single modality. In practical tasks, the target object is often represented by heterogeneous multi-modal features. One of the most direct ways to extend the single-modal hashing method to multi-modal scenarios is to cascade multi-modal features and then use the cascaded features as the input of the single-modal model. This simple processing method will lead to some thorny problems, such as the curse of dimensionality and information redundancy. To overcome this problem, multi-modal hashing methods have been developed to explore the complementary information between multi-modal data.

[0003] Multimodal hashing methods focus on fusing multiple modalities to learn compact hash codes. Paired multimodal query instances are encoded according to a trained multimodal hash function. Recently, some impressive multimodal hashing methods have been proposed to support large-scale multimodal retrieval. Multi-feature hashing (MFH) simultaneously preserves the individual structural information of each modality and the global structural information of all modalities. Multi-view latent hashing (MvLH) adopts a unified kernel feature space to learn multimodal common latent factors. Efficient parameter-free adaptive multimodal hashing (EPAMH) adaptively captures modality changes and preserves discriminative semantic information in the multimodal feature space. Although these methods have clever modeling ideas, they do not achieve significant retrieval performance because high-level semantic information is not exploited in the model. To address this issue, online multimodal hashing with dynamic query adaptation (OMH-DQ) is proposed to adaptively learn modality weights through a supervised self-weighted fusion strategy. OMH-DQ achieves impressive retrieval performance, but it contains many hyperparameters that need to be determined in optimization, which greatly hinders its promotion in practical application scenarios. Considering this, Yu et al. proposed a lightweight model called Adaptive Multimodal Fusion Hashing (AMFH) to learn hash codes by exploiting the discriminative column properties of the Hadamard matrix. However, AMFH transforms the optimization problem of a signed function into a relaxation problem, which brings inevitable information loss. Summary of the invention

[0004] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a multimodal hash retrieval method guided by a Hadamard matrix, which jointly performs subspace learning and hash learning in an overall framework. On the one hand, the proposed method adaptively captures the complementary information between multimodal data. In addition, the multimodal space is reconstructed by an integrated matrix decomposition method to enhance the feature representation capability. On the other hand, the present method utilizes the column attributes of the Hadamard matrix to generate the target code for each category. Soft binary constraints are used on the target binary matrix to improve the discriminability of the model and solve the problems raised in the background technology.

[0005] The present invention provides the following technical solutions:

[0006] A multimodal hash retrieval method guided by a Hadamard matrix comprises the following steps:

[0007] S1, establishing training data, wherein the training data includes a plurality of different modal data and corresponding category information, and the different modal data of the training data share the same semantic information;

[0008] S2, establishing a common subspace and projecting the multimodal data into the common subspace, where the multimodal data are adaptively fused to capture complementary information between the multimodal data;

[0009] S3, introduce the Hadamard matrix, and generate the target hash center of each category through the discriminative column properties of the Hadamard matrix;

[0010] S4, Hadamard matrix guided multimodal fusion hash,generation algorithm;

[0011] S5, establish a query data, calculate its nonlinear representation characteristics, obtain optimal parameters according to the algorithm, use a hash function to query the query data for multimodal query, and generate a query hash code.

[0012] Preferably, in step S1, the mth mode of the training data is represented as where d m is the feature dimension, n is the number of training sets, and each training data is represented as Among them l i is the class label, represented by a unified hash code B∈{-1,+1} c×n To represent multimodal instances, c is the length of the hash code. In order to maintain the correlation within the modality, the nonlinear transformation representation of each modality is calculated through the anchor sample.

[0013] Preferably, in the anchor point sample calculation, for the mth mode, in is the set of anchor samples, σ m is the Gaussian kernel parameter, and the transformation feature of the mth mode is expressed as The expression 1 is as follows:

[0014]

[0015] in, α m ≥0,μ m ≥0, γ is the smoothing parameter.

[0016] Preferably, in step S2, the original modality data is reconstructed by an integrated matrix decomposition method to enhance the representation capability of the fused features.

[0017] Preferably, the target representation matrix of the training data is represented as H∈{+1,-1} c×n , where c = 2 k , k is a positive integer, H is limited to binary code. In order to give it more degrees of freedom, the binary constraint is relaxed to a soft constraint, that is, Z = H + H⊙Q, where Q ≥ 0. An orthogonal transformation matrix R is introduced to reduce the difference between the common subspace and the target feature space. The expression 2 is as follows:

[0018]

[0019] Among them, Q ≥ 0, RR T =I, ⊙ represents element-by-element product.

[0020] Preferably, by combining equation 1 and equation 2, the overall objective function is obtained, and the expression equation 3 for solving the overall objective function is as follows:

[0021]

[0022] in, α m ≥0,μ m ≥0,Q≥0,RR T =I, where β and γ are two penalty parameters.

[0023] Preferably, an alternating direction multiplier method is used to solve equation 3, and the method is repeated one or more times until the objective function equation 3 converges to a stable value.

[0024] Preferably, the alternating direction multiplier method is used to solve equation 3, and the detailed steps are as follows:

[0025] A: In formula 3, other variables remain unchanged and W is updated m , let the objective function be W mThe derivative of (m=1, 2, ..., M) is zero, and the equation 4 is as follows:

[0026] B: Other variables remain unchanged, update U m , the objective function is about U m The derivative of (m=1, 2, ..., M) is set to zero, and the equation 5 is as follows:

[0027] C: Other variables remain unchanged, update V, let Z = H + H⊙Q, and get formula 6 as follows:

[0028]

[0029] D: other variables remain unchanged, update R, and get formula 7 as follows:

[0030]

[0031] Make VZ T =U∑V T , we get formula eight as follows: R = UV T ;

[0032] E: other variables remain unchanged, update Q, let C = RV-H, and get formula 9 as follows:

[0033]

[0034] Solving the optimal solution of equation 9, we get equation 10 as follows: Q = max(H⊙C, 0);

[0035] F: other variables remain unchanged, update μ m , retaining the same m The related terms are as follows:

[0036] in, The Lagrangian function of equation 11 is shown in equation 12:

[0037]

[0038] Find the optimal μ m , we get formula 13 as follows:

[0039]

[0040] in,

[0041] G: other variables remain unchanged, update α m , similarly, we get formula 14 as follows:

[0042]

[0043] H: Repeat the process from step A to step G until the objective function equation 3 converges to a stable value, and summarize equation 3 in the algorithm.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] (1) The present invention provides a Hadamard matrix guided multimodal hash retrieval method, which jointly performs subspace learning and hash learning in an overall framework. On the one hand, the proposed method adaptively captures the complementary information between multimodal data. In addition, the multimodal space is reconstructed through an integrated matrix decomposition method to enhance the feature representation capability. On the other hand, the method utilizes the column attributes of the Hadamard matrix to generate the target code for each category, and adopts soft binary constraints on the target binary matrix to improve the discriminability of the model.

[0046] (2) The present invention provides a Hadamard matrix-guided multimodal hash retrieval method, which combines feature learning and hash learning into a joint learning framework through an adaptive multimodal fusion hash method. The learned hash code can capture the complementary information and category semantic information between multimodal data, thereby improving the retrieval performance.

[0047] (3) The present invention provides a multi-modal hash retrieval method guided by a Hadamard matrix, which adopts soft binary constraints on the target hash code allocated according to the Hadamard matrix to enhance the discrimination ability of the hash function to be learned. At the same time, the orthogonal transformation matrix is ​​used for hash learning to reduce quantization loss and improve the quality of the hash code. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 The present invention is a flow chart of the method.

[0050] Figure 2 This is a framework diagram of the method of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Example 1

[0054] See also Figure 1-2 As shown, a multimodal hash retrieval method guided by a Hadamard matrix includes the following steps:

[0055] S1, establishing training data, wherein the training data includes a plurality of different modal data and corresponding category information, and the different modal data of the training data share the same semantic information;

[0056] S2, establishing a common subspace and projecting the multimodal data into the common subspace, where the multimodal data are adaptively fused to capture complementary information between the multimodal data;

[0057] S3, introduce the Hadamard matrix, and generate the target hash center of each category through the discriminative column properties of the Hadamard matrix;

[0058] S4, Hadamard matrix guided multimodal fusion hash,generation algorithm;

[0059] S5, establish a query data, calculate its nonlinear representation characteristics, obtain optimal parameters according to the algorithm, use a hash function to query the query data for multimodal query, and generate a query hash code.

[0060] Given a new query instance in S5 First, calculate its nonlinear representation characteristics According to the optimal parameters obtained by Algorithm 1, use the hash function For multimodal query i Generates a hash code.

[0061] Multimodal data are projected into a common subspace, where they are adaptively fused to capture complementary information between them. To enhance the representation of fused features, the original modal features are reconstructed by integrating matrix decomposition methods. Hadamard matrices are introduced to generate target hash centers for each category. To improve the discriminative power of the method, soft binary constraints are applied to the target codes of the training data. In the target space, multimodal data with the same category tend to be close to their common target features.

[0062] Based on the above observations, the Hadamard matrix guided multimodal hashing method (HGMH) is proposed to solve the above problems. This method combines feature learning and hash learning into a unified framework (the framework of the HGMH method is as follows Figure 2 As shown). First, a common low-dimensional subspace is found to effectively fuse high-dimensional multimodal features. In order to improve the representation ability of the fused features, an integrated matrix decomposition method is used to reconstruct the multimodal space. An orthogonal transformation matrix is ​​introduced to reduce quantization loss. Secondly, the discriminative column properties of the Hadamard matrix are used to generate the hash center of each category. Our model drives data of the same category to migrate to their common hash center. In order to enhance the discriminative ability of the hash model to be learned, soft binary constraints are adopted for the target hash code in the proposed HGMH framework.

[0063] In step S1, the mth mode of the training data is represented as where d m is the feature dimension, n is the number of training sets, and each training data is represented as Among them l i is the class label, represented by a unified hash code B∈{-1,+1} c×n To represent multimodal instances, c is the length of the hash code. In order to maintain the correlation within the modality, the nonlinear transformation representation of each modality is calculated through the anchor sample.

[0064] In the anchor point sample calculation, for the mth mode, in is the set of anchor samples, σ m is the Gaussian kernel parameter, and the transformation feature of the mth mode is expressed as The expression 1 is as follows:

[0065]

[0066] in, α m ≥0,μ m ≥0, γ is the smoothing parameter.

[0067] In step S2, the original modality data is reconstructed by integrating the matrix decomposition method to enhance the representation ability of the fused features.

[0068] The target representation matrix of the training data is represented as H∈{+1,-1} c×n , where c = 2 k , k is a positive integer, H is limited to binary code. In order to give it more degrees of freedom, the binary constraint is relaxed to a soft constraint, that is, Z = H + H⊙Q, where Q ≥ 0. An orthogonal transformation matrix R is introduced to reduce the difference between the common subspace and the target feature space. The expression 2 is as follows:

[0069]

[0070] Among them, Q ≥ 0, RR T =I, ⊙ represents element-by-element product.

[0071] By combining equation 1 and equation 2, we get the overall objective function. The expression equation 3 for solving the overall objective function is as follows:

[0072]

[0073] in, α m ≥0,μ m ≥0,Q≥0,RR T =I, where β and γ are two penalty parameters.

[0074] A. Model

[0075] Assume that the given training data contains multiple different modalities and their corresponding category information, and the mth modality is represented as where d m and n are the feature dimension and the number of training sets respectively. Different modalities in each instance share the same semantic information, and each instance is represented as Among them l i is the class label. This method aims to learn a unified hash code B∈{-1,+1} c×n To represent multimodal instances, c is the length of the hash code. In order to maintain the correlation within the modality, the nonlinear transformation representation of each modality is first calculated through the anchor sample. Specifically, for the mth modality, in is the set of anchor samples, σ m is the Gaussian kernel parameter. The transformation feature of the mth mode is expressed as This method finds a common subspace to fully represent multimodal instances. In this common space, complementary multiple modalities are adaptively fused. In addition, a reconstruction term is added to the model to enhance the representation ability of subspace features. The above idea is expressed as shown in Expression 1.

[0076] In the Hamming space, samples with the same category are induced to be close to the common hash center. As an effective hash center generation method, the Hadamard matrix has been verified to be discriminative in hash learning. In the present invention, the Hadamard matrix is ​​introduced to generate target codes for multimodal data of different categories.

[0077] The target representation matrix of the entire training data is represented as H∈{+1,-1} c×n , where c = 2 k , k is a positive integer. H is strictly limited to binary codes. In order to give it more degrees of freedom, the strict binary constraint is relaxed to a soft constraint, i.e., Z = H + H⊙Q, where Q ≥ 0. In fact, the relaxed target code increases the spacing between classes, thus improving the discrimination ability of this model. An orthogonal transformation matrix R is introduced to reduce the difference between the common subspace and the target feature space. The expression is shown in Equation 2.

[0078] By combining Equation 1 and Equation 2, we can obtain the overall objective function, as shown in Equation 3.

[0079] Formula 3 is a non-convex problem, and it is difficult to solve the objective function directly. The alternating direction multiplier method is used to solve Formula 3, and it is repeated once or multiple times until the objective function Formula 3 converges to a stable value.

[0080] The alternating direction multiplication method is used to solve equation 3, and the detailed steps are as follows:

[0081] A: In formula 3, other variables remain unchanged and W is updated m , let the objective function be W m The derivative of (m=1, 2, ..., M) is zero, and the equation 4 is as follows:

[0082] B: Other variables remain unchanged, update U m , the objective function is about U m The derivative of (m=1, 2, ..., M) is set to zero, and the equation 5 is as follows:

[0083] C: Other variables remain unchanged, update V, let Z = H + H⊙Q, and get formula 6 as follows:

[0084]

[0085] D: other variables remain unchanged, update R, and get formula 7 as follows:

[0086]

[0087] Make VZ T =U∑V T , we get formula eight as follows: R = UV T ;

[0088] E: other variables remain unchanged, update Q, let C = RV-H, and get formula 9 as follows:

[0089]

[0090] Solving the optimal solution of equation 9, we get equation 10 as follows: Q = max(H⊙C, 0);

[0091] F: other variables remain unchanged, update μ m , retaining the same m The related terms are as follows:

[0092] in, The Lagrangian function of equation 11 is shown in equation 12:

[0093]

[0094] Find the optimal μ m , we get formula 13 as follows:

[0095]

[0096] in,

[0097] G: other variables remain unchanged, update α m , similarly, we get formula 14 as follows:

[0098]

[0099] H: Repeat the process from step A to step G until the objective function equation 3 converges to a stable value, and summarize equation 3 in the algorithm.

[0100] Example 2

[0101] On the basis of Example 1, in step S5, the query data is trained in the algorithm, and the detailed steps are as follows:

[0102] a: Input: training set The class vector L = {l1, ..., ln} generates the target code H by using the Hadamard matrix;

[0103] b: Compute nonlinear representation

[0104] c: Initialize V, Q, R, α m and μ m ;

[0105] d: Update W according to formula 4 m (m=1, ..., M);

[0106] e: Update U according to formula 5 m (m=1, ..., M);

[0107] f: Update V according to formula 6;

[0108] g: Update R according to formula 8;

[0109] h: Update Q according to formula 10;

[0110] i: Update μ according to formula 13 m (m=1, ..., M);

[0111] j: Update d according to formula 14 m (m=1, ..., M);

[0112] k: repeat step dj;

[0113] l: until convergence, output: W m , U m , α m .

[0114] Complexity Analysis

[0115] The computational complexity of the algorithm is O(np), and the computational cost of nonlinear features is O(np). In the algorithm, update W m , U m ,V,R,Q,μ m and α m It takes O(np 2 )、O(npc)、O(npc)、O(nc 2 ), O(ncp), and O(ncp), so the time complexity of each iteration is O(np 2 ), the total time complexity of the method of the present invention is O(iter×np 2 ), where iter is the number of iterations.

[0116] Example 3

[0117] On the basis of Example 2, experiments were conducted to verify the effectiveness of the method of the present invention.

[0118] A. Dataset

[0119] Some comparative experiments are conducted on three benchmark datasets: Pascal Sentence, LabelMe, and MIR Flickr to verify the effectiveness of the proposed method. The detailed descriptions of these datasets are as follows.

[0120] Pascal Sentence is a subset of VOC 2008, consisting of 1,000 image-text pairs divided into 20 categories. Each image is described by 5 separate sentences. CNN visual features

[28] and 100-dimensional probability vectors are used to represent images and text, respectively. For fair comparison, 800 image-text pairs are randomly selected as the training dataset (40 pairs per category) and 100 image-text pairs are randomly selected as the test dataset (5 pairs per category).

[0121] LabelMe contains 2,688 external scenes from eight categories. 245-dimensional phrase frequency and 512-dimensional GIST features are used to represent text samples and image samples, respectively. In our experiments, a subset of 2016 image-text pairs is randomly selected as the training dataset, and the remaining 672 image-text pairs are used as the test dataset.

[0122] MIRFlickr contains 20,015 images and their associated text labels collected from the Flickr website. Each image-text pair sample is classified into multiple categories. Each image is represented by a 4,096-dimensional visual feature vector extracted from VGGNet pre-trained on the ImageNet dataset. 1,386-dimensional bag-of-words features are used to represent the text samples. The entire dataset is randomly divided into a training set containing 5,000 image-text pairs, a retrieval set containing 17,772 image-text pairs, and a query set containing 2,243 image-text pairs.

[0123] B. Experimental Setup

[0124] In the experiments, the proposed method is compared with eight hashing methods, including ITQ, LSH, DLLE, HCOH, MFH, MVLH, OMH-DQ, and AMFH. ITQ, LSH, DLLE, and DLLE are four representative single-mode hashing methods. For fair comparison, multi-mode features are concatenated for single-mode hashing methods. MFH, MVLH, OMH-DQ, and AMFH are four state-of-the-art hashing methods.

[0125] The hyperparameters of the baselines are tuned within their original ranges. For single-label datasets, such as PascalSentence and LabelMe, the columns of the Hadamard matrix are set to the hash centers of different classes respectively. Following AMFH, the centroids of multiple class hash centers are used as the data target codes for the multi-label MIRFlickr dataset. γ is empirically set to 0.1, 0.1, and 0.3 on PascalSentence, LabelMe, and MIRFlickr, respectively. The values ​​of β and λ are tuned in the candidate range of {1e-5, 1e-3, 1e-1, 1, 1e 2}, and the best results are reported in this paper. In the comparative experiments, the hash code length is manually set to 32 bits, 64 bits, 128 bits, and 256 bits. Mean average precision (mAP) and precision-recall (PR) are used as evaluation metrics to evaluate all methods. The maximum number of iterations during the optimization process is set to 100. All experiments were performed on a workstation with Matlab R2018b, 16GB memory and Intel(R) Core(TM) i7-10700 CPU@2.90GHz.

[0126] C. Results and Discussion

[0127] Comparative experiments were conducted on Pascal Sentence, LabelMe, and MIRFlickr when the hash code length was set to 32 bits, 64 bits, 128 bits, and 256 bits, respectively. The experimental results are shown in Table 1, and it can be seen that the proposed method consistently outperforms all the compared methods. Specifically, compared with the best single-mode hashing method, the proposed method achieves an average improvement of 28.86%, 29.38%, and 3.93% on Pascal Sentence, LabelMe, and MIRFlickr, respectively. The results show that the proposed method outperforms the single-mode hashing method for multimodal retrieval. The possible reason is that the multimodal feature cascade method causes information redundancy, and the single-mode hashing method does not fully exploit the complementary information between multimodal. The proposed method achieves further improvement on the multi-label MIRFlickr than on the single-label dataset, because the generation process of the hash center of multi-label data brings a large quantization loss by calculating the centroid of multiple hash centers. The proposed method is 1.1%, 0.9%, and 1.5% higher than the best multimodal baseline on Pascal Sentence, LabelMe, and MIRFlickr, respectively.

[0128] The comparison results show that the method of jointly performing subspace learning and hash learning is very effective in encoding multimodal data. Compared with the previous work (AMFH), the method proposed in this paper can achieve better performance. The results show that the relaxed target code can enhance the discriminative ability of the hash function to be learned and improve the retrieval accuracy.

[0129] It can be seen that the method is very competitive on the MIRFlickr dataset. On pascal Sentence and LabelMe, the method achieves comparable results compared with OMH-DQ and AMFH. The experimental results on the three datasets show that the proposed method of learning the hash function by relaxing the target hash code can further improve the retrieval performance. In order to explore the effectiveness of the relaxation method, the following experiments are designed: Let 'Non-relaxing' mean that the method does not introduce soft constraints. In other words, we always set the Q matrix to zero in the optimization. Comparative experiments were conducted on three datasets, and the experimental results are shown in Table II. We can see that the soft constraint method outperforms the "non-relaxing" method. The experimental results show that the method with soft constraints can promote more discriminative hash learning.

[0130]

[0131] Table 1: mAP comparison of multimodal retrieval at different positions

[0132]

[0133] Table 2: Comparison results between the non-relaxed method and the proposed soft constraint method

[0134] D. Parameter Sensitivity Analysis

[0135] The influence of two hyperparameters on the performance of the method is explored. The values ​​of β and λ are adjusted in the candidate range {1e -5, 1e -3, 1e -1, 1, 1e 2} to observe the performance changes of the model. The performance changes of β on Pascal Sentence, LabelMe and MIRFlickr show that when the value of β is greater than 1e -1, the performance of the method is superior and stable.

[0136] A multimodal hashing method based on Hadamard matrix is ​​used for large-scale multimodal retrieval, called Hadamard matrix guided multimodal hashing (HGMH). HGMH assigns a target hash center to each category through the Hadamard matrix, and then introduces a soft binary constraint on the target matrix of the training data to increase the class distance. The multimodal data are projected into a common subspace to learn the fused feature representation. In addition, in order to enhance the unified feature representation capability, an integrated matrix decomposition method is used to reconstruct the multimodal space. In the model, an orthogonal transformation matrix is ​​used to reduce the difference between the common subspace and the target feature space. Comprehensive experimental results on three benchmark multimedia datasets demonstrate the superiority of the HGMH method.

[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and modifications. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal hash retrieval method guided by Hadamard matrix, It is characterized in that The following steps are involved: S1, establishing training data, wherein the training data includes a plurality of different modal data and corresponding category information, and the different modal data of the training data share the same semantic information; S2, establishing a common subspace and projecting the multimodal data into the common subspace, where the multimodal data are adaptively fused to capture complementary information between the multimodal data; S3, introduce the Hadamard matrix, and generate the target hash center of each category through the discriminative column properties of the Hadamard matrix; S4, Hadamard matrix guided multimodal fusion hash,generation algorithm; S5, establishing a query data, calculating its nonlinear representation features, obtaining optimal parameters according to the algorithm, using a hash function to query the query data for multimodal query, and generating a query hash code; In step S1, the mth mode of the training data is represented as where d m is the feature dimension, n is the number of training sets, and each training data is represented as Among them l i is the class label, represented by a unified hash code B∈{-1,+1} c×n To represent multimodal instances, c is the length of the hash code. In order to maintain the correlation within the modality, the nonlinear transformation representation of each modality is calculated through the anchor point sample; In the anchor point sample calculation, for the mth mode, in is the set of anchor samples, σ m is the Gaussian kernel parameter, and the transformation feature of the mth mode is expressed as The expression 1 is as follows: in, α m ≥0,μ m ≥0, γ is the smoothing parameter; The target representation matrix of the training data is represented as H∈{+1,-1} c×n , where c = 2 k , k is a positive integer, H is limited to binary codes. In order to give it more degrees of freedom, the binary constraint is relaxed to a soft constraint, that is, Z = H + H⊙Q, where Q ≥ 0. An orthogonal transformation matrix R is introduced to reduce the difference between the common subspace and the target feature space. The expression 2 is as follows: Among them, Q ≥ 0, RR T =I,⊙ represents element-by-element product; By combining equation 1 and equation 2, we get the overall objective function. The expression equation 3 for solving the overall objective function is as follows: in, α m ≥0,μ m ≥0,Q≥0,RR T =I, where β and γ are two penalty parameters.

2. According to the multimodal hash retrieval method guided by the Hadamard matrix as claimed in claim 1, It is characterized in that In step S2, the original modality data is reconstructed by integrating the matrix decomposition method to enhance the representation ability of the fused features.

3. According to the multimodal hash retrieval method guided by the Hadamard matrix of claim 1, It is characterized in that The alternating direction multiplier method is used to solve Equation 3, and this is repeated one or more times until the objective function Equation 3 converges to a stable value.