A multimodal retrieval method based on feature fusion hash algorithm

Abstract

This paper proposes a multimodal retrieval method based on a feature fusion hashing algorithm. The method comprises the following steps: first, obtaining a multimodal training set and extracting features from each modality; second, calculating low-dimensional modality-specific features of each modality's features through PCA; third, learning joint features of the features of each modality through sparse projection; and calculating the orthogonal rotation matrix and fusion coefficient of the low-dimensional modality-specific features and joint features respectively; fourth, calculating the hash code of the query sample based on the orthogonal rotation matrix and fusion coefficient; and finally, calculating the Hamming distance between the hash code of the query sample and the hash code of the database instance, and selecting data with a Hamming distance less than a threshold as the retrieval result. The method can mine the shared semantic information and modality-specific internal structure information between multimodal data information to learn discriminative hash codes. It effectively solves the discrete optimization problem without generating large quantization errors due to relaxation.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal retrieval, and in particular to a multimodal retrieval method based on a feature fusion hash algorithm. Background Art

[0002] In recent years, the explosive growth of multimedia data, such as images, text, audio, and video, has magnified the challenges in the field of information search. Low search efficiency has become one of the most pressing issues in large-scale multimedia applications. Hashing, as an effective representation tool, has attracted increasing attention due to its advantages of low storage cost and high processing speed. For example, if 128 bits are used to represent visual objects, one million images only require 16MB of memory to store. The similarity between two images is measured by an efficient exclusive OR (XOR) operation, which greatly improves search efficiency.

[0003] Hashing methods aim to transform the original space into a compact binary space. Most hashing methods focus on the unimodal image scenario of retrieving related images from an image database given an image. Among unimodal hashing methods, discrete locally linear embedding hashing (DLLH), similarity-preserving link hashing (SPLH), and neighborhood-preserving hashing (NPH) have achieved good image retrieval performance. Cross-modal hashing methods learn multiple hash functions to generate hash codes for multiple different modalities. However, existing unimodal and cross-modal hashing methods are unable to encode query and database data described by heterogeneous multimodal features.

[0004] In practice, in some real-world applications, such as sensor networks and content submission systems with mandatory options, data instances are often represented comprehensively using multiple modalities. Different modalities describe instances from different perspectives, and this data is referred to as multimodal data. Multimodal hashing methods have been proposed to support large-scale multimodal data retrieval. Supervised multimodal hashing has achieved significant progress in retrieval performance. However, supervised methods typically require high-quality labels to train their models, which is impractical. In contrast, unsupervised multimodal hashing methods avoid using supervised information and instead learn hash codes by leveraging the distribution and topological structure of the data. Despite the remarkable success of representative unsupervised multimodal hashing methods, many challenges remain unresolved. These methods project multimodal data into a common latent subspace to learn unified hash features. The learned common features represent the shared semantic properties of the multimodal data. While these methods preserve the shared semantic properties of multimodal data, they do not explore the unique properties of each modality in their models. Indeed, due to their distinct physical structures, different modalities are highly diverse. Modality-specific feature learning can preserve the intra-modal energy and specific structural properties of each modality. Therefore, how to retain both shared semantic properties and unique attribute information in the learned hash code deserves further study. Summary of the Invention

[0005] In response to the deficiencies in the above-mentioned background technology, the present invention proposes a multimodal retrieval method based on a feature fusion hash algorithm, which not only learns a unified common subspace for different modalities to retain their common semantic attributes, but also uses principal component analysis (PCA) to find an independent subspace for each modality to obtain the specific attribute information of each modality; it solves the existing technical problem that the shared semantic attributes and unique attribute information cannot be retained simultaneously in the hash code to be learned.

[0006] The technical solution of the present invention is achieved as follows:

[0007] A multimodal retrieval method based on feature fusion hash algorithm, the steps of which are as follows:

[0008] Step 1: Obtain a multimodal training set and extract features of each modality separately;

[0009] Step 2: Calculate the low-dimensional modality-specific features of each modality through PCA;

[0010] Step 3: Learn the joint features of the features of each modality through sparse projection;

[0011] Step 4: Calculate the orthogonal rotation matrix and fusion coefficient of the low-dimensional specific modal features and joint features respectively;

[0012] Step 5: Calculate the hash code of the query sample based on the orthogonal rotation matrix and fusion coefficient;

[0013] Step 6: Calculate the Hamming distance between the hash code of the query sample and the hash code of the database instance, and take the data with a Hamming distance less than the threshold as the retrieval result.

[0014] Preferably, the multimodal training set includes n pairs of M modalities, represents the characteristic matrix of the mth mode, where Indicates dimension n×d m The set of real numbers, d m is the feature dimension, m∈{1,…,M}, and n is the number of training samples.

[0015] Preferably, the low-dimensional specific modal feature is: V m =X m A m , where V m ∈R n×c Represents the feature matrix of the mth mode after PCA processing, R n×c A represents a set of real numbers with dimension n×c, where c is the length of the hash code. m is the mapping matrix obtained by PCA calculation, X m Represents the characteristic matrix of the mth mode.

[0016] Preferably, the joint feature is expressed as:

[0017]

[0018] in, represents the projection matrix of the mth modality, V0 is the joint feature matrix, ||·|| F represents the Frobenius norm of the matrix, ||·|| 2,1 l represents a vector 2,1 norm, λ m is the adaptive parameter, β m is the balance parameter and γ is the smoothing parameter.

[0019] Preferably, the orthogonal rotation matrix is ​​expressed as:

[0020]

[0021] in, B∈{-1,1} c×n , α t >0,V t ={V0,V m |m∈{1,…,M}},R tis the rotation matrix of the t-th view, α t is the weight coefficient of the t-th view.

[0022] Preferably, the fusion coefficient is obtained by:

[0023] The joint matrix and the orthogonal rotation matrix are reorganized into the overall objective function, which is expressed as:

[0024]

[0025] The alternating optimization method is used to optimize and solve the overall objective function:

[0026] Fixed other variables update P m : By retaining m Related items, get about P m The sub-function is:

[0027]

[0028] Apply formula (5) to P m The partial derivative of is set to zero and the optimal P is calculated. m :

[0029]

[0030] in, is a diagonal matrix, Indicates P m The i-th row of

[0031] Fix other variables to update V0: delete items not related to V0 and sub-functions about V0:

[0032]

[0033] Let the partial derivative of formula (7) with respect to V0 be zero, we can get:

[0034]

[0035] Fixed other variables to update R t : By treating other variables as constants, t The relevant sub-functions are:

[0036]

[0037]

[0038] Transform (9) to obtain its equivalent formula:

[0039]

[0040] Through the matrix BV t The optimal R is obtained by singular value decomposition t =UQ, where U and Q are the left and right singular vectors, respectively;

[0041] Fixing other variables and updating B: By keeping the terms related to B, the subfunction of minimizing B is equivalent to maximizing the subfunction:

[0042]

[0043] The closed-form solution for B is as follows:

[0044]

[0045] Fixed other variables update α t : with α t The relevant sub-functions are as follows:

[0046]

[0047] Submission in, From this we can get α t The closed-form solution is:

[0048]

[0049] Fix other variables and update λ m : About λ m The subfunction is written as follows:

[0050]

[0051] Submission in, Get λ m The optimal solution is as follows:

[0052]

[0053] The above sub-functions are updated alternately, and the entire alternating optimization process is iteratively executed until the overall objective function converges to a stable value and the optimal λ is learned from the training phase. m ,α t ,P m ,R t .

[0054] Preferably, the hash code of the query sample is expressed as:

[0055]

[0056] Among them, H q The query sample x is generatedq The hash code of is the query sample x q The multimodal features of , sgn(·) represents the sign function, α0 represents the weight of the joint features in the fusion process, A t Represents the projection matrix corresponding to the t-th mode.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1) A novel unsupervised multimodal hashing algorithm is proposed to jointly perform specific feature and unified feature learning, and to collaboratively preserve common semantic information and individual modality information into hash codes;

[0059] 2) We developed an efficient alternating optimization method for the Feature Fusion Hashing (JIFFH) model, which has high learning efficiency due to the direct closed-form solution of discrete variables.

[0060] 3) JIFFH is compared with the most advanced unsupervised multimodal hashing methods on public multimodal datasets, highlighting the superiority of the JIFFH algorithm in multimodal retrieval tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 Flowchart of the present invention.

[0063] Figure 2 This is a block diagram of the feature fusion hash algorithm of the present invention.

[0064] Figure 3 The intra-class compactness of IFH, JFH, and JIFFH; (a) IFH, (b) JFH, (c) JIFFH.

[0065] Figure 4 The sensitivity analysis diagram of the parameter β in the example of the present invention on three datasets; (a) MIRFlickr, (b) NUS-WID, (c) MS COCO.

[0066] Figure 5 This is a sensitivity analysis diagram of the parameter γ in the example of the present invention on three data sets.

[0067] Figure 6Convergence curves on MIRFlickr, NUS-WID, and MS COCO when the hash code length is fixed at 24 bits; (a) MIRFlickr, (b) NUS-WID, (c) MS COCO. DETAILED DESCRIPTION

[0068] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0069] like Figure 1 As shown in the figure, a multimodal retrieval method based on feature fusion hashing algorithm, the feature fusion hashing algorithm (JIFFH) adaptively captures the complementary information between multiple modalities to learn a unified representation feature. Through principal component analysis, the specific modal feature information is extracted to preserve the structure and data energy within the modality. The method adopts a self-weighting strategy and can adaptively find the combination weight to combine the joint features of multimodal data and the specific modal features. Figure 2 As shown in Figure 1, the algorithm consists of two parts: an offline training phase and an online hash coding phase. During the offline training phase, modality-specific features are calculated using PCA, and joint features are obtained through sparse projection learning. The modality-specific features and shared joint features are adaptively weighted and fused to obtain the final hash code. During the online hash coding phase, hash codes are generated for multimodal samples based on the learned joint and modality-specific feature fusion module. The specific steps are as follows:

[0070] Step 1: Obtain a multimodal training set and extract the features of each modality separately; the multimodal training set contains n pairs of M modalities, represents the characteristic matrix of the mth mode, where Indicates dimension n×d m The set of real numbers, d m is the feature dimension, m∈{1,…,M}, n is the number of training samples. B∈{-1,+1} c×n is the hash code matrix to be learned, where c is the hash code length. The goal of the method proposed in this invention is to learn the mapping function from the original multimodal space to the discrete Hamming space. In this invention, the image modality and text modality are used as examples to describe the multimodal learning problem, that is, M = 2. Symbol ||·|| F Represents the Frobenius norm of the matrix. Q∈R n×m The l2-norm of is defined as:

[0071]

[0072] Step 2: Calculate the low-dimensional specific modal features of each modality through PCA; heterogeneous modalities have different physical structures, so it is necessary to retain the specific attributes of each modality. To deal with this problem, the present invention learns the specific feature representation of each independent modality to retain its specific attribute information. The principal component analysis (PCA) projection matrix is ​​introduced to generate specific modal features that retain the main energy of each modality. The low-dimensional specific modal features are: V m =X m A m , where V m ∈R n ×c Indicates the m The feature matrix of the modes after PCA processing, R n×c A represents a set of real numbers with dimension n×c, where c is the length of the hash code. m is the mapping matrix obtained by PCA calculation, X m Represents the characteristic matrix of the mth mode.

[0073] Step 3: Learn the joint features of the features of each modality through sparse projection; different modalities in a multimodal instance describe the same semantic content. In other words, paired images and texts have the same semantic information. In order to maintain the consistency of semantic information between different modalities, the multimodal data is projected into a unified latent semantic space to obtain a joint feature representation. The joint feature representation is:

[0074]

[0075] in, represents the projection matrix of the mth modality, V0 is the joint feature matrix, ||·|| F represents the Frobenius norm of the matrix, ||·|| 2,1 represents the l2-norm of the vector. The l2-norm constraint is to enhance the discriminability of projection learning; m is the adaptive parameter, β m is the balance parameter and γ is the smoothing parameter.

[0076] Step 4: Calculate the orthogonal rotation matrix and fusion coefficient of the low-dimensional specific modal features and joint features respectively;

[0077] In existing work, orthogonal rotation matrices are often used to reduce binary quantization loss. The general theoretical expression is:

[0078]

[0079] Make it obey R T R=I c×c, B∈{-1, 1} c×n , where R∈R c×c is an orthogonal rotation matrix, and V represents a low-dimensional representation matrix. The present invention fuses multimodal features so that the learned multimodal hash code incorporates rich complementary information from multiple modalities. In real data scenarios, there are differences between different view features, so the present invention adopts an adaptive weighting mode to capture the complementary information between multi-view data. The orthogonal rotation matrix used in the present invention is expressed as:

[0080]

[0081] in, B∈{-1,1} c×n , α t >0,V t ={V0,V m |m∈{1,…,M}},R t is the rotation matrix of the t-th view, α t is the weight coefficient of the t-th view.

[0082] The method for obtaining the fusion coefficient is:

[0083] The joint matrix and the orthogonal rotation matrix are reorganized into the overall objective function, which is expressed as:

[0084]

[0085] The alternating optimization method is used to optimize and solve the overall objective function:

[0086] Fixed other variables update P m : By retaining m Related items, get about P m The sub-function is:

[0087]

[0088] Apply formula (5) to P m The partial derivative of is set to zero and the optimal P is calculated. m :

[0089]

[0090] in, is a diagonal matrix, Indicates P m The i-th row of

[0091] Fix other variables to update V0: delete items not related to V0 and sub-functions about V0:

[0092]

[0093] Let the partial derivative of formula (7) with respect to V0 be zero, we can get:

[0094]

[0095] Fixed other variables to update R t : By treating other variables as constants, t The relevant sub-functions are:

[0096]

[0097]

[0098] Transform (9) to obtain its equivalent formula:

[0099]

[0100] Through the matrix BV t The optimal R is obtained by singular value decomposition t =UQ, where U and Q are the left and right singular vectors, respectively;

[0101] Fixing other variables and updating B: By keeping the terms related to B, the subfunction of minimizing B is equivalent to maximizing the subfunction:

[0102]

[0103] The closed-form solution for B is as follows:

[0104]

[0105] Fixed other variables update α t : with α t The relevant sub-functions are as follows:

[0106]

[0107] Submission in, From this we can get α t The closed-form solution is:

[0108]

[0109] Fix other variables and update λ m : About λ m The subfunction is written as follows:

[0110]

[0111] Submission in, Get λ m The optimal solution is as follows:

[0112]

[0113] The above sub-functions are updated alternately, and the entire alternating optimization process is iteratively executed until the overall objective function converges to a stable value and the optimal λ is learned from the training phase. m ,α t ,P m ,R t .

[0114]

[0115]

[0116] The present invention analyzes the computational complexity of the training phase. The computational cost of the specific modal feature of the mth modality is O(nd m c) In each iteration, update P m ,V0,R t The time complexities of and are O(nc 2 ) and O(nc 2 ). In general, when n>d m When >c, the time complexity of the method of the present invention is O(iter×nd 2 ), where d = max(d m |m∈{1,…,M}), iter represents the number of iterations.

[0117] Step 5: Calculate the hash code of the query sample based on the orthogonal rotation matrix and the fusion coefficient; the hash code of the query sample is expressed as:

[0118]

[0119] Among them, H q The query sample x is generated q The hash code of is the query sample x q The multimodal features of , sgn(·) represents the sign function, α0 represents the weight of the joint features in the fusion process, A t Represents the projection matrix corresponding to the t-th mode.

[0120] Step 6: Calculate the Hamming distance between the hash code of the query sample and the hash code of the database instance, and take the data with a Hamming distance less than the threshold as the retrieval result.

[0121] Experimental verification

[0122] Extensive experiments are conducted on three benchmark datasets, MIRFlickr, NUS-WIDE, and MS COCO, to evaluate the performance of the proposed JIFFH. All datasets consist of both image and text modalities. The detailed experimental setup is described as follows:

[0123] (1) MIRFlickr consists of 20,015 images and their text labels from the Flickr website. Each image-text pair sample is labeled with multiple categories. Each image is represented by a 4,096-dimensional visual feature vector extracted from VGGNet pre-trained on the ImageNet dataset. 1,386-dimensional bag-of-words features are used to represent the text samples. The entire dataset is randomly divided into a training set containing 5,000 image-text pairs, a retrieval set containing 17,772 image-text pairs, and a query set containing 2,243 image-text pairs.

[0124] (2) NUS-WIDE contains 195,834 image-text samples and is a real-world web multimodal dataset. Each image-text sample is classified into at least one of 21 categories. VGGNet is used to extract 4,096-dimensional features for each image. Each text is represented by a 1,000-dimensional bag-of-words feature. In the experiment, 5,000 image-text samples and 2,085 image-text samples were randomly selected as the training set and query set, respectively. The remaining 193,749 samples were used as the retrieval set.

[0125] (3) MS COCO consists of 18,000 training image-text pairs, a retrieval subset of 82,783 pairs, and a test set of 5,981 pairs. The dataset includes 80 object categories, and each image-text pair belongs to at least one of these categories. Each image is represented by a 4,096-dimensional CNN feature vector extracted from VGGNet pre-trained on the ImageNet dataset, and each text is represented by a 2,000-dimensional bag-of-words feature. The statistics of the three datasets are shown in Table 1.

[0126] Table 1 Statistics of the three datasets

[0127]

[0128] In the experiments, the proposed JIFFH is compared with the following seven state-of-the-art unsupervised multimodal hashing methods to verify the effectiveness of the proposed method. Specifically,

[0129] PCAH learns a binarization function by concatenating multimodal features into a unified feature vector and then performing principal component analysis (PCA).

[0130] ITQ first finds a low-dimensional space of the same dimension as the Hamming space to be learned based on PCA, and then applies an orthogonal transformation to reduce the quantization error.

[0131] MFH preserves the local structure information within each individual modality, while the global local structure of all modality data is also preserved in the hash code.

[0132] MAH learns a compact representation that reveals hidden semantic information while fusing multi-modal information via regularized kernel non-negative matrix factorization.

[0133] MVLH learns binary encoding from a unified kernel feature space and adaptively learns the weights of different modalities through reconstruction error.

[0134] MvDH jointly performs matrix factorization and non-negative spectral clustering to embed multimodal data into compact hash codes.

[0135] EPAMH is an unsupervised parameter-free multimodal hashing learning model that adaptively learns modal weights to capture modality variations.

[0136] There are several parameters in the JIFFH model: γ, β m . Where γ is a smoothing parameter that controls the weight change. In the experiments on MIRFlickr, NUS WIDE and MS COCO, γ is set to -1, 5 and 5 respectively. The present invention sets a candidate set {1e -4 ,1e -3 ,1e -2 ,1e -1 ,10,1e 3}, and adjust β within the candidate range m When setting {β1=1e 3 ,β2=1e 3}, set {β1=10,β2=1e on NUS-WIDE 3}, set {β1=1e -2 ,β2=1e -4 The best results were recorded when}. We empirically set the hash code length to 8, 16, 24, and 32 bits, respectively; the maximum number of iterations was set to 50. The mean average precision (mAP) was used to evaluate our method. All experiments were run on a workstation equipped with Matlab R2018b, 16GB of memory, and an Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz.

[0137] In this experiment, comparative experiments were conducted on the MIR Flickr, NUS-WIDE, and MS COCO datasets to evaluate the performance of our method. The experimental results are shown below.

[0138] 1) Results on MIRFlickr: The mAP results of JIFFH and baselines on the MIRFlickr dataset are reported in Table 2. It can be observed that the proposed JIFFH outperforms all baselines. Specifically, the average performance of the proposed method is 2.7% higher than the best baseline. The performance of the proposed method further improves with the increase of hash code length. Compared with EPAMH, the proposed method is consistently better in retrieval accuracy when the hash code length is any length in the experimental setting. Experimental results show that the proposed method is very effective for learning hashes.

[0139] Table 2: mAP results of all methods with different bit lengths on MIRFlickr

[0140]

[0141] 2) Results on NUS-WIDE: On the NUS-WIDE dataset, the mA results of all methods under different hash code length settings are shown in Table 3. It can be found that the proposed method is always better than the baseline. When the hash code length is set to 32 bits, the proposed method is 5.29% higher than the best baseline method EPAMH. Compared with the best single-mode hash method ITQ, the proposed method achieves an average improvement of 5.15% under all hash length settings. The experimental results show that the JIFFH proposed in this paper can enhance the representation ability of the hash code, and the proposed feature fusion method can effectively improve the retrieval performance.

[0142] Table 3: mAP results of all methods with different bit lengths on NUS-WIDE

[0143]

[0144]

[0145] 3) Results on MS COCO: Table 4 shows the mAP results of all methods on the MS COCO dataset. It can be seen that the method proposed in this paper achieves the highest retrieval accuracy compared to other methods. The retrieval accuracy of the method proposed in this paper is approximately 2.6% higher than that of EPAMH. The performance changes very smoothly with increasing hash code length. This may be because longer encoding lengths may carry redundant information, which does not significantly improve retrieval performance. This means that the method proposed in this paper can achieve better retrieval performance on the MS COCO dataset even when the encoding code length is shorter.

[0146] Table 4: mAP results of all methods with different bit lengths on MS COCO.

[0147]

[0148] 4) Ablation study: The JIFFH proposed in this invention combines specific modal features and joint multimodal features to learn a robust hash function. In order to verify the effectiveness of the proposed framework, the following ablation experiments were designed. On the one hand, only the specific modal feature learning branch is considered to learn the hash, and this learning method is named the acronym "IFH". On the other hand, the model is trained only by the joint feature learning branch of the framework proposed in this invention, and the learning process is represented by the abbreviation "JFH". Comparative experiments were carried out on three datasets, and the experimental results are shown in Table 5. It can be clearly observed that JIFFH achieves the highest retrieval accuracy compared with IFH and JFH. In addition, the t-SNE tool is used to visualize the distribution of the learned binary features on the single-label WiKi dataset. Figure 3 (a) Figure 3 (b) and Figure 3 (c) shows the intra-class compactness of IFH, JFH, and JIFFH, respectively. It can be clearly seen that the proposed JIFFH can learn the most discriminative hash code features compared to IFH and JFH. Ablation experiment results show that the shared and specific properties of multimodal data can jointly promote multimodal hash learning.

[0149] Table 5: Ablation experiment results on three datasets

[0150]

[0151]

[0152] The influence of the hyperparameters in the model on the retrieval results is analyzed experimentally. The model proposed in this invention needs to determine the γ and β before training the model. m By changing γ and β in the candidate range m The performance of the proposed model is observed by using the value of . Figure 4 and Figure 5 The model proposed by the present invention is shown in Figure 2. m and γ performance changes. Figure 4 As shown in , it can be found that the performance of the model proposed by the present invention varies slightly within a large range of combinations of β1 and β2. Figure 5 In

[15] , it is clearly observed that the best results are achieved when γ is set to -1, 5, and 5 on the MIRFlickr, NUS-WIDE, and MS COCO datasets, respectively.

[0153] The convergence of Algorithm 1 on three datasets is verified through experiments. When the hash code length is fixed to 24 bits, the convergence curves of MIRFlickr, NUS-WIDE and MS COCO are as follows: Figure 6 As shown. The trend of the convergence curve for other hash code lengths is similar to Figure 6 The experimental results show that the objective function of the proposed JIFFH method is monotonically decreasing on the three datasets and converges to a stable value within 20 iterations.

[0154] This paper develops a new multimodal hashing method for multimodal retrieval, also known as Joint Feature and Modality-Specific Feature Fusion Hashing (JIFFH). In the proposed JIFFH, the semantic information shared between multimodal data information and the modality-specific intra-structural information are mined to learn discriminative hash codes. In addition, the discrete optimization problem is effectively solved without large quantization errors due to relaxation. Comparative experiments are conducted on three public multimodal datasets. The experimental results show that the proposed JIFFH outperforms the state-of-the-art multimodal methods on large-scale multimodal retrieval tasks.

[0155] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal retrieval method based on feature fusion hash algorithm, characterized in that: The steps are as follows: Step 1: Obtain a multimodal training set and extract features of each modality separately; Step 2: Calculate the low-dimensional modality-specific features of each modality through PCA; Step 3: Learn the joint features of the features of each modality through sparse projection; Step 4: Calculate the orthogonal rotation matrix and fusion coefficient of the low-dimensional specific modal features and joint features respectively; The joint feature is expressed as: in, represents the projection matrix of the mth modality, V0 is the joint feature matrix, ||·|| F represents the Frobenius norm of the matrix, ||·|| 2,1 l represents a vector 2,1 norm, λ m is the adaptive parameter, β m is the balance parameter, γ is the smoothing parameter; X m represents the feature matrix of the mth mode; M represents the total number of modes; c is the hash code length, d m is the characteristic dimension; The orthogonal rotation matrix is ​​expressed as: in, B∈{-1,1} c×n , α t >0,V t ={V0,V m |m∈{1,…,M}},R t is the orthogonal rotation matrix of the t-th view, α t is the weight coefficient of the t-th view; The fusion coefficient is obtained by reorganizing the joint matrix and the orthogonal rotation matrix into an overall objective function, which is expressed as: The alternating optimization method is used to optimize the overall objective function and obtain the optimal fusion coefficient And the projection matrix P m and the orthogonal rotation matrix R t ; Step 5: Calculate the hash code of the query sample based on the orthogonal rotation matrix and fusion coefficient; Step 6: Calculate the Hamming distance between the hash code of the query sample and the hash code of the database instance, and take the data with a Hamming distance less than the threshold as the retrieval result.

2. The multimodal retrieval method based on feature fusion hash algorithm according to claim 1 is characterized in that: The multimodal training set contains n pairs of M modalities. represents the characteristic matrix of the mth mode, where Indicates dimension n×d m is a set of real numbers, m∈{1,…,M}, and n is the number of training samples.

3. The multimodal retrieval method based on feature fusion hash algorithm according to claim 2 is characterized in that: The low-dimensional specific modal features are: V m =X m A m , where V m ∈R n×c Represents the feature matrix of the mth mode after PCA processing, R n×c A represents a set of real numbers with dimension n×c, m is the mapping matrix obtained by PCA calculation.

4. The multimodal retrieval method based on feature fusion hash algorithm according to any one of claims 1 to 3, characterized in that: The method of optimizing the overall objective function using the alternating optimization method is as follows: fix other variables and update the projection matrix P m :By preserving the projection matrix P m Related items, get the projection matrix P m The sub-function is: Apply equation (5) to the projection matrix P m The partial derivative of is set to zero and the optimal P is calculated. m : in, is a diagonal matrix, Represents the projection matrix P m The i-th row of Fix other variables and update the joint feature matrix V0: delete items that are not related to the joint feature matrix V0. The subfunction of the joint feature matrix V0 is: Let the partial derivative of formula (7) with respect to the joint feature matrix V0 be zero, and we get: Fix other variables and update the orthogonal rotation matrix R t : By treating other variables as constants, the orthogonal rotation matrix R t The relevant sub-functions are: Transform (9) to obtain its equivalent formula: Through the matrix BV t The optimal R is obtained by singular value decomposition t =UQ, where U and Q are the left and right singular vectors, respectively; Fix other variables and update B: By keeping the terms related to B, the subfunction of minimizing B is equivalent to maximizing the subfunction: The closed-form solution for B is as follows: Fixed other variables update weight coefficient α t : and weight coefficient α t The relevant sub-functions are as follows: Submission in, The weight coefficient α is obtained from this t The closed-form solution is: Fix other variables and update the adaptive parameter λ m :About the adaptive parameter λ m The subfunction is written as follows: Submission in, Get λ m The optimal solution is as follows: The above sub-functions are updated alternately, and the entire alternating optimization process is iteratively executed until the overall objective function converges to a stable value and the optimal λ is learned from the training phase. m ,α t ,P m ,R t .

5. The multimodal retrieval method based on feature fusion hash algorithm according to claim 4 is characterized in that: The hash code of the query sample is expressed as: Among them, H q The query sample x is generated q The hash code of is the query sample x q The multimodal features of , sgn(·) represents the sign function, α0 represents the weight of the joint features in the fusion process, A t Represents the projection matrix corresponding to the t-th mode.

Images