Multi-modal semantic fusion Hash method based on self-paced learning cooperating with Fisher criterion

Abstract

The invention belongs to the technical field of multi-modal Hash, and discloses a multi-modal semantic fusion Hash method based on self-paced learning and Fisher criterion, which comprises the following steps of: mining semantic consistency association of multi-source heterogeneous data, introducing a self-paced learning mechanism, and constructing a multi-source heterogeneous semantic fusion Hash model based on self-paced learning weight; the multi-source heterogeneous data content is processed, analyzed and understood step by step, and it is ensured that the model is learned step by step from easy to difficult. Meanwhile, the Fisher criterion is embedded to enhance the fusion discrimination of different semantic categories, and the influence of semantic similarity on multi-source heterogeneous semantic fusion is further considered on the basis of increasing the correlation and potential structure information between boundaries of different categories and concerned data. Therefore, the semantic fusion Hash code with high-quality discriminant power is generated. According to the method, massive multi-source heterogeneous data are effectively processed to obtain valuable information, the expression capability of semantic fusion hash coding is enhanced, and the robustness and flexibility of an algorithm model are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal fusion, and in particular relates to a multimodal semantic fusion hashing method of self-paced learning and collaborative Fisher criterion. Background Art

[0002] Today, with the highly developed network information technology, multi-source heterogeneous data in multimedia has shown explosive growth, and people are entering a new era of intelligent data at an unprecedented speed. With the rapid growth of large-scale multi-source heterogeneous data, the high-dimensional feature space of massive multi-source heterogeneous data has a series of problems such as information redundancy, large representation differences, insufficient feature reliability, and a semantic gap between low-level and high-level features, which increases the difficulty of semantic fusion analysis of multi-source heterogeneous data. At the same time, there are massive fragmented heterogeneous data in multimedia, which are usually accompanied by problems such as noise and semantic ambiguity, and there are complex coupling relationships between different modalities. Existing multi-source heterogeneous semantic fusion hash learning methods are difficult to process these massive heterogeneous data with high quality. In the research of supervised multi-source heterogeneous semantic fusion hashing algorithms, the semantic similarity between data points is usually captured using label semantic information, which generates more discriminative binary hash codes. For example, the orthogonal basis proposed in the literature (Liu X., Wang XZ, Cheung YMFDDH: Fast Discriminative Discrete Hashing for Large-Scale Cross-Modal Retrieval) introduces fast discriminative discrete hashing, which returns the target hash code of the training sample to the corresponding semantic label, thereby maximizing and clearly capturing the discriminative ability of semantic information. The literature (Sun Y., Ren ZW, Hu P., et al. Hierarchical Consensus Hashing for Cross-Modal Retrieval) proposed a hierarchical consensus cross-modal hashing method, which adopted a coarse-to-fine hierarchical hashing scheme and consensus learning to efficiently encode data into the consensus space in a progressive manner, thereby gradually reducing the semantic gap. When the above method processes large-scale complex multi-source heterogeneous data, due to the inconsistent semantic difficulty of the training samples, the conventional matrix decomposition learning model is used to learn the semantic fusion hash code, which is very likely to lead to large fusion errors. At the same time, many supervised methods often directly reference the label matrix in the common subspace, but the learned hash code is of poor quality and limited universality. Summary of the invention

[0003] To solve the above technical problems, the present invention provides a multimodal semantic fusion hashing method based on self-paced learning and Fisher criterion. By mining the semantic consistency association of multi-source heterogeneous data and referencing the self-paced learning weighting mechanism, a self-paced learning weighted multi-source heterogeneous sample decomposition model is constructed, so that the hash learning model can gradually process, analyze and understand the content of multi-source heterogeneous data, ensuring that the model can use the difficulty of the samples to learn from easy to difficult step by step; at the same time, by embedding the Fisher criterion to enhance the fusion discrimination of different semantic categories, by increasing the boundaries of different categories and focusing on the correlation and potential structural information between data, the influence of semantic similarity on multi-source heterogeneous semantic fusion is further considered, thereby generating high-quality discriminative semantic fusion hash codes.

[0004] The multimodal semantic fusion hashing method of the self-paced learning and Fisher criterion of the present invention comprises the following steps:

[0005] Step 1: Obtain large-scale multi-source heterogeneous data, and use a deep network model to extract sample feature information of multimodal data; use a Gaussian mapping function to perform kernelization processing on the sample feature information to obtain the kernelized multimodal data features;

[0006] Step 2: Using the self-paced learning strategy and the Fisher discriminant criterion to build a multimodal semantic fusion hash learning model, and using the alternating iterative optimization solution method to solve the optimal solution of the multimodal semantic fusion hash learning model parameters;

[0007] Step 3: Using the optimal solution obtained in step 2, the enhanced projection function learning model is integrated to learn the projection function of projecting multimodal data features to hash codes;

[0008] Step 4: Based on the multimodal data projection function, perform semantic fusion hash coding processing on the newly arrived multimodal data to generate corresponding semantic fusion hash codes.

[0009] Furthermore, step 1 is specifically as follows:

[0010] Step 1-1: Obtain large-scale multi-source heterogeneous data, including image data, text data, and label data, and use the deep network model to extract the image sample feature information and text sample feature information of the multimodal data to obtain X t ∈R dt×n is the data matrix of the tth modality, X1 represents the image modality data matrix, X2 represents the text modality data matrix, d t is the feature dimension of the t-th mode, n represents the number of samples; Q1∈R d×r is the mapping matrix of the image modality, Q2∈R d×ris the mapping matrix of the text modality, where d is the feature dimension and r is the potential feature matrix dimension; the potential feature matrix of the image modality is represented by V1∈R r×n , the latent feature matrix of the text modality is represented as V2∈R r×n .

[0011] The category label matrix Y∈{y1,y2,…y n} has binary restrictions; is the similarity matrix, whose value is in ||.||2 represents the two-norm, ||.|| F represents the F-norm;

[0012] Step 1-2: Use radial basis function RBF to preprocess image sample data and text sample data, where δ(.) is a Gaussian kernel projection function, and then obtain the nonlinear feature matrix δ(X1) of the image sample data and the nonlinear feature matrix δ(X2) of the text sample data.

[0013] Furthermore, step 2 is specifically as follows:

[0014] Step 2-1: In order to improve the hash learning model's ability to gradually process, analyze and understand multi-source heterogeneous data content, ensure that the model can use the difficulty of the sample to gradually learn from easy to difficult, and build a self-learning weighted multi-source heterogeneous sample decomposition model:

[0015]

[0016]

[0017] Where Z1 is the weighted diagonal matrix of the image sample, which represents the difficulty of learning the image sample features. is the element value on the diagonal of the ith row of Z1, and Z2 is the weighted diagonal matrix of the text sample, which represents the difficulty of learning the features of the text sample. is the value of the element on the diagonal of the ith row of Z2, diag(.) is the diagonal matrix representation, and the parameter η controls the order in which the image and text samples are introduced. U is the instance weight matrix, which represents the reliability of learning, so that the model starts from simple instances and gradually learns more difficult instances; is the value of the element on the diagonal of the i-th row of U;

[0018] Step 2-2: The self-paced learning weighted multi-source heterogeneous sample decomposition model strategy of step 2-1 is further coordinated with the Fisher discriminant criterion to construct the global optimization function corresponding to the multimodal semantic fusion hash learning model:

[0019]

[0020] stB∈{-1,1},V1V1 T =nI r ,V2V2 T =nI r ,V11 n =0 r ,V21 n =0 r ,R T R=I r ,

[0021]

[0022] Where Β is the hash code matrix. D1 is the image mapping matrix from the label semantics projected to the image potential feature space, and D2 is the text mapping matrix from the label semantics projected to the text potential feature space. W is a non-negative relaxation matrix. Due to the restriction of W ≥ 0, the distances between different categories of Y are all increased after relaxation optimization, in order to make the model more flexible in the process of dealing with classification problems; C is a predefined constant matrix, whose elements are determined by the elements in the category label matrix Y. When Y ij =1, C ij = +1, when Y ij = -1, C ij =-1; T is the label relaxation matrix, T i represents the label relaxation matrix element of the i-th class, is the average value of all samples in the i-th class; M i is the sum of all n in T i The same average column vector; Ω(D1,D2,T) represents the regularization of matrices D1, D2 and T, μ is the balance parameter between label semantics and latent feature space, β is the balance parameter between image latent feature space and text latent feature space, θ is the semantic shift parameter, σ is the Fisher criterion parameter, γ is the regularization parameter, and ⊙ is the Hadamard product of the matrix;

[0023] Step 2-3: Use the alternating iterative optimization solution method to obtain the optimal solution for the parameters of the multimodal semantic fusion hash learning model.

[0024] Furthermore, in step 2-3, an alternating iterative optimization solution method is used, that is, when solving a closed-form solution of a parameter, the parameter is regarded as a variable, and other variables except the parameter are fixed, specifically:

[0025] Step 2-3-1, when updating the optimization parameter Q1, fix the other parameters except parameter Q1, and solve the closed-form solution of Q1 as:

[0026] Q1=(Z1T Z1δ(X1)UU T V1 T )(V1UU T V1 T Z1 T Z1) -1 ;

[0027] Step 2-3-2, when updating the optimization parameter Q2, fix the other parameters except parameter Q2, and solve the closed-form solution of Q2 as:

[0028] Q2=(Z2 T Z2δ(X1)UU T V2 T )(V2UU T V2 T Z2 T Z2) -1 ;

[0029] Step 2-3-3: When updating the optimization parameter R, fix all parameters except parameter R and adjust the matrix Perform singular value decomposition to obtain the left orthogonal matrix A1 and the right orthogonal matrix A2, and then solve the closed-form solution of R as follows:

[0030]

[0031] Step 2-3-4: When updating the optimization parameter D1, fix the other parameters except parameter D1, and solve the closed-form solution of D1 as follows:

[0032]

[0033] Step 2-3-5, when updating the optimization parameter D2, fix the other parameters except parameter D2, and solve the closed-form solution of D2 as:

[0034]

[0035] Step 2-3-6, when updating the optimization parameter T, fix the other parameters except parameter T, and solve the closed-form solution of T as follows:

[0036] T=(μD1 T D1+μD2 T D2+θI+2σI+λI) -1

[0037]

[0038] Step 2-3-7, when updating the optimization parameter W, fix the other parameters except parameter W, and solve the closed-form solution of W as:

[0039] W=max(C⊙(TY),0);

[0040] Step 2-3-8, when updating the optimization parameter V1, other parameters except parameter V1 are fixed, and the global optimization function corresponding to the multimodal semantic fusion hash learning model is converted to:

[0041]

[0042] stB∈{-1,1},V1V1 T =nI r ,V11 n =0 r ,R T R=I r ,

[0043]

[0044] In order to facilitate the explanation of the optimal closed-form solution process for V1, Γ1 is called the intermediate matrix transfer variable of the image, and J1 is the characteristic decomposition transfer variable of the image. For the matrix Γ1J1Γ1 T Perform eigendecomposition to obtain the eigendecomposition left matrix E1 and the eigendecomposition right matrix F1. is the random orthogonal matrix corresponding to the right matrix of the eigendecomposition, is the random orthogonal matrix corresponding to the left matrix of the eigendecomposition, then the optimal closed-form solution of V1 is

[0045] Step 2-3-9, when updating the optimization parameter V2, other parameters except parameter V2 are fixed, and the global optimization function corresponding to the multimodal semantic fusion hash learning model is converted to:

[0046]

[0047] stB∈{-1,1},V2V2 T =nI r ,V21 n =0 r ,R T R=I r

[0048] In order to facilitate the explanation of the optimal closed-form solution process for V2, Γ2 is called the intermediate matrix transfer variable of the text, and J2 is the eigendecomposition transfer variable of the text. For the matrix Γ2J2Γ2 T Perform eigendecomposition to obtain the eigendecomposition left matrix E2 and the eigendecomposition right matrix F2. is the random orthogonal matrix corresponding to the right matrix of the eigendecomposition, is the random orthogonal matrix corresponding to the left matrix of the eigendecomposition, then the optimal closed-form solution of V2 is

[0049] Step 2-3-10, for the sake of convenience, t = 1 and t = 2; then when updating the optimization parameter Z t When the fixed parameter Z is removed t Other parameters except Z t The closed-form solution is:

[0050]

[0051] In the formula, Represents the quantization error of the i-th feature, and assigns weights to each feature based on this error That is, each feature will have a corresponding weight value;

[0052] Step 2-3-11, when updating the optimization parameter U, fix the other parameters except parameter U, and solve the closed-form solution of parameter U as follows:

[0053]

[0054] In the formula, It represents the loss function of the i-th instance, and then solves the weights of all instances

[0055] Step 2-3-12, when updating the optimization parameter B, fix the other parameters except parameter B, and solve the closed-form solution of B as follows:

[0056]

[0057] Where sign(.) is the sign function. When the input parameter value is positive, its value is 1; otherwise, its value is -1.

[0058] Furthermore, step 3 is specifically as follows:

[0059] Adopt multimodal semantic fusion projection function learning model:

[0060]

[0061] Where ρ is the balance parameter of the projection function, H t is the projection function of the t-th mode. The above formula is optimized to solve the projection function H of multimodal data features projected to the hash code. t The optimal closed-form solution of is:

[0062] H t =(BB T +ρI) -1 (lBL TLδ(X t ) T +ρBδ(X t ) T )(δ(X t )δ(X t ) T +γI) -1

[0063] Furthermore, step 3 is specifically as follows:

[0064] Adopt multimodal semantic fusion projection function learning model:

[0065]

[0066] In the formula, μ is the balance parameter of the projection function, γ is the weight parameter, and H t is the projection function of the t-th mode, and the formula is optimized to solve the projection function H of multimodal data features projected to the hash code t The optimal closed-form solution of is:

[0067] H t =(BB T +μI) -1 (lBL T Lδ(X t ) T +ηBδ(X t ) T )(δ(X t )δ(X t ) T +γI) -1 .

[0068] Furthermore, step 4 is specifically as follows: using the multimodal data projection function H trained and learned in step 3 t , perform RBF preprocessing on the newly arrived multimodal data to obtain Gaussian kernel projection features Perform semantic fusion hash coding processing; solve the semantic fusion hash coding B o The optimal closed-form solution is:

[0069]

[0070] The beneficial effects of the present invention are:

[0071] 1) The present invention mines the semantic consistency of multimodal data and introduces a label semantic relaxation matrix, so that the binary restrictions of the category label matrix are relaxed, and the fusion potential feature matrix learned by the algorithm model has stronger generalization ability and accuracy; at the same time, by incorporating orthogonal constraints, the orthogonal constraints help to reduce the redundant information between multimodal sample features, ensuring that multimodal sample features can provide independent and valuable information, while also improving the efficiency of the optimization solution of the multimodal semantic fusion hash learning algorithm; by increasing the correlation and potential structural information between different category boundaries and attention data, semantic similarity is also incorporated into the multimodal semantic fusion hash learning process, and an alternating iterative optimization solution method is used to solve the optimal solution of the multimodal semantic fusion hash learning model parameters, thereby generating a semantic fusion hash code with high-quality discriminative power.

[0072] 2) In order to enable the multimodal semantic fusion hash learning model to gradually process, analyze and understand multi-source heterogeneous data content, the algorithm model of the present invention incorporates a self-paced learning strategy to ensure that the model learns gradually from easy to difficult. In order to enable the algorithm model to maximize the semantic differences between samples of different categories and the semantic differences between samples of the same category, the multimodal semantic fusion hash learning model embeds the Fisher criterion to enhance the fusion discrimination of different semantic categories. The present invention constructs a multimodal semantic fusion hash learning method through self-paced learning and Fisher criterion, so that the algorithm model can effectively process massive multi-source heterogeneous data to obtain valuable information, strengthen the representation ability of semantic fusion hash coding, and improve the robustness and flexibility of the algorithm model. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0074] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0075] like Figure 1 As shown, the multimodal semantic fusion hash learning method of the present invention based on self-paced learning and Fisher criterion includes the following steps:

[0076] Step 1, multimodal data acquisition and processing stage: obtain large-scale multi-source heterogeneous data, and use the deep network model to extract the image sample feature information and text sample feature information of the multimodal data. Since the sample feature information of the extracted multimodal data is linearly inseparable, a Gaussian mapping function is used for kernelization processing to map the sample feature information to a high-dimensional space, so that it is linearly separable in the high-dimensional space, and the multimodal data features after kernelization are obtained;

[0077] Step 2, model parameter solution training phase: In order to ensure that the constructed algorithm model can gradually analyze and understand the multi-source heterogeneous data content from easy to difficult, and at the same time, in order to further enable the algorithm model to maximize the semantic differences of samples of different categories and the semantic differences of samples of the same category; for this purpose, a multimodal semantic fusion hash learning model is constructed with a self-paced learning strategy and Fisher discriminant criterion, and an alternating iterative optimization solution method is used to solve the optimal solution of the multimodal semantic fusion hash learning model parameters;

[0078] Step 3, projection function training and solution phase: using the optimal solution of the multimodal semantic fusion hash learning model parameters obtained in step 2, the fusion enhanced projection function learning model is used to learn the projection function of projecting multimodal data features into hash codes;

[0079] Step 4, semantic fusion hash coding stage: Use the multimodal data projection function trained and learned in step 3 to perform semantic fusion hash coding on the newly arrived multimodal data to generate corresponding high-quality and reliable semantic fusion hash codes.

[0080] The method of the present invention is further illustrated by the following examples.

[0081] Step 1 is as follows:

[0082] Step 1-1: The present invention uses the VGG16 network model to extract features from image modality samples in the multimodal dataset to obtain image features. d1=512 is the feature dimension of the image mode, and n=10000 is the size of the multimodal dataset. The bag-of-words model is used to train and learn text data to obtain text features. d2=2912 is the feature dimension of the text modality. Q1∈R d×r is the mapping matrix of the image modality, Q2∈R d×r is the mapping matrix of the text modality, where d=1500 is the feature dimension, r is the potential feature matrix dimension, and the values ​​set in the present invention are 32, 64, and 128 respectively; the potential feature matrix of the image modality is expressed as V1∈R r×n , the latent feature matrix of the text modality is represented as V2∈R r×n . The category label matrix Y∈{y1,y2,…y n} has binary restrictions; is the similarity matrix, whose value is in represents the two-norm, ||.|| F represents the F-norm.

[0083] Step 1-2, since linear functions cannot capture the nonlinear manifold structure between heterogeneous modalities well, in practical applications, it is necessary to focus on the linear inseparability of samples when learning hash coding; and the kernelization method can handle the linear inseparability problem of most heterogeneous data, and is often used to capture the nonlinear characteristics of heterogeneous data. Therefore, the present invention uses radial basis function (RBF) to preprocess sample data, and the number of randomly selected sample anchor points in the image modality and text modality is 1500, where δ(.) is a Gaussian kernel projection function, and the nonlinear feature matrix δ(X1) of the image sample data and the nonlinear feature matrix δ(X2) of the text sample data are obtained.

[0084] Step 2: Model parameter calculation and training phase:

[0085] Step 2-1: In order to improve the hash learning model's ability to gradually process, analyze and understand multi-source heterogeneous data content, ensure that the model can use the difficulty of the sample to gradually learn from easy to difficult, and build a self-learning weighted multi-source heterogeneous sample decomposition model:

[0086]

[0087]

[0088] Where Z1 is the weighted diagonal matrix of the image sample, which represents the difficulty of learning the image sample features. is the element value on the diagonal of the ith row of Z1, and Z2 is the weighted diagonal matrix of the text sample, which represents the difficulty of learning the features of the text sample. is the element value on the diagonal of the i-th row of Z2, diag(.) is the diagonal matrix representation, and the parameter η controls the order in which the image and text samples are introduced. U is the instance weight matrix, which represents the reliability of learning; is the value of the element on the diagonal of the i-th row of U.

[0089] Step 2-2: To ensure that the constructed algorithm model can analyze and understand the multi-source heterogeneous data content from easy to difficult, and to further maximize the semantic differences of samples of different categories and the semantic differences of samples of the same category, the self-learning weighted multi-source heterogeneous sample decomposition model strategy of step 2-1 is further coordinated with the Fisher discriminant criterion to construct the global optimization function corresponding to the multimodal semantic fusion hash learning model:

[0090]

[0091] stB∈{-1,1},V1V1 T =nI r ,V2V2T =nI r ,V11 n =0 r ,V21 n =0 r ,R T R=I r ,

[0092]

[0093] Where Β is the hash code matrix; D1 is the image mapping matrix from the label semantics projected to the image latent feature space; D2 is the text mapping matrix from the label semantics projected to the text latent feature space; W is the non-negative relaxation matrix; C is a predefined constant matrix whose elements are determined by the elements in the category label matrix Y. When Y ij =1, C ij = +1, when Y ij = -1, C ij =-1; T is the label relaxation matrix, T i represents the label relaxation matrix element of the i-th class, is the average value of all samples in the i-th class; M i is the sum of all n in V i The same average column vector; Ω(D1, D2, T) represents the regularization of matrices D1, D2 and T; μ is the balance parameter between label semantics and potential feature space, and the setting value of this invention is 1e 0 β is the balance parameter between the image potential feature space and the text potential feature space, and the setting value of the present invention is 1e 3 θ is the semantic offset parameter, and the value set in the present invention is 1e 3 ; σ is the Fisher criterion parameter, and the setting value of the present invention is 1e -3 ; γ is the regularization parameter, and the value set in this invention is 1e -2 ; ⊙ is the Hadamard product of the matrix.

[0094] Step 2-3: In order to optimize and solve the parameters of the multimodal semantic fusion hash learning model efficiently and quickly, the present invention adopts an alternating iterative optimization solution method, that is, when solving the closed-form solution of a certain parameter, the parameter is regarded as a variable, and other variables except the parameter are fixed.

[0095] Step 2-3-1, when updating the optimization parameter Q1, fix the other parameters except parameter Q1, and solve the closed-form solution of Q1 as:

[0096] Q1=(Z1 T Z1δ(X1)UU T V1 T )(V1UU T V1T Z1 T Z1) -1

[0097] Step 2-3-2, when updating the optimization parameter Q2, fix the other parameters except parameter Q2, and solve the closed-form solution of Q2 as:

[0098] Q2=(Z2 T Z2δ(X1)UU T V2 T )(V2UU T V2 T Z2 T Z2) -1

[0099] Step 2-3-3: When updating the optimization parameter R, fix all parameters except parameter R and adjust the matrix Perform singular value decomposition to obtain the left orthogonal matrix A1 and the right orthogonal matrix A2, and then solve the closed-form solution of R as follows:

[0100]

[0101] Step 2-3-4: When updating the optimization parameter D1, fix the other parameters except parameter D1, and solve the closed-form solution of D1 as follows:

[0102]

[0103] Step 2-3-5, when updating the optimization parameter D2, fix the other parameters except parameter D2, and solve the closed-form solution of D2 as:

[0104]

[0105] Step 2-3-6, when updating the optimization parameter T, fix the other parameters except parameter T, and solve the closed-form solution of T as follows:

[0106] T=(μD1 T D1+μD2 T D2+θI+2σI+λI) -1

[0107]

[0108] Step 2-3-7, when updating the optimization parameter W, fix the other parameters except parameter W, and solve the closed-form solution of W as:

[0109] W=max(C⊙(TY),0)

[0110] Step 2-3-8, when updating the optimization parameter V1, other parameters except parameter V1 are fixed, and the global optimization function corresponding to the multimodal semantic fusion hash learning model is converted to:

[0111]

[0112] stB∈{-1,1},V1V1 T =nI r ,V11 n =0 r ,R T R=I r ,

[0113]

[0114] In order to facilitate the explanation of the optimal closed-form solution process for V1, Γ1 is called the intermediate matrix transfer variable of the image, and J1 is the characteristic decomposition transfer variable of the image. For the matrix Γ1J1Γ1 T Perform eigendecomposition to obtain the eigendecomposition left matrix E1 and the eigendecomposition right matrix F1. is the random orthogonal matrix corresponding to the right matrix of the eigendecomposition, is the random orthogonal matrix corresponding to the left matrix of the eigendecomposition. Then the optimal closed-form solution of V1 is

[0115] Step 2-3-9, similarly, when updating the optimization parameter V2, other parameters except parameter V2 are fixed, and the global optimization function corresponding to the multimodal semantic fusion hash learning model is converted to:

[0116]

[0117] stB∈{-1,1},V2V2 T =nI r ,V21 n =0 r ,R T R=I r

[0118] In order to facilitate the explanation of the optimal closed-form solution process for V2, Γ2 is called the intermediate matrix transfer variable of the text, and J2 is the eigendecomposition transfer variable of the text. For the matrix Γ2J2Γ2 T Perform eigendecomposition to obtain the eigendecomposition left matrix E2 and the eigendecomposition right matrix F2. is the random orthogonal matrix corresponding to the right matrix of the eigendecomposition, is the random orthogonal matrix corresponding to the left matrix of the eigendecomposition. Then the optimal closed-form solution of V2 is

[0119] Step 2-3-10, for the sake of convenience, t = 1, 2. Then when updating the optimization parameter Z t When the fixed parameter Z is removed t Other parameters except Z t The closed-form solution is:

[0120]

[0121] In the formula, Represents the quantization error of the i-th feature, and assigns weights to each feature based on this error That is, each feature will have a corresponding weight value.

[0122] Step 2-3-11, when updating the optimization parameter U, fix the other parameters except parameter U, and solve the closed-form solution of parameter U as follows:

[0123]

[0124] In the formula, It represents the loss function of the i-th instance, and then the weights of all instances can be solved

[0125] Step 2-3-12, when updating the optimization parameter B, fix the other parameters except parameter B, and solve the closed-form solution of B as follows:

[0126]

[0127] Where sign(.) is the sign function. When the input parameter value is positive, its value is 1; otherwise, its value is -1.

[0128] In the projection function training and solving phase of step 3, since the linear regression projection function mapping method cannot deeply explore the nonlinear structure of the sample when mapping the features of the original data to the low-dimensional Hamming space, in order to further enhance the learning ability of the projection function, a multimodal semantic fusion projection function learning model is adopted:

[0129]

[0130] Where ρ = 1e -2 is the balance parameter of the projection function, H t is the projection function of the t-th mode. The above formula is optimized to solve the projection function H of multimodal data features projected to the hash code. t The optimal closed-form solution of is:

[0131] H t =(BB T +ρI) -1(lBL T Lδ(X t ) T +ρBδ(X t ) T )(δ(X t )δ(X t ) T +γI) -1

[0132] The semantic fusion hash coding stage in step 4: using the multimodal data projection function H trained and learned in step 3 t , perform RBF preprocessing on the newly arrived multimodal data to obtain Gaussian kernel projection features The semantic fusion hash coding process is performed, and the present invention solves the semantic fusion hash coding B o The corresponding optimal closed-form solution is:

[0133]

[0134] The semantic fusion hash code B is obtained through the above process. o .

[0135] Through simulation experiments, the retrieval accuracy mAP values ​​of various comparison methods on the NUS-WIDE dataset are:

[0136] Table 1 mAP values ​​of various comparison methods on the NUS-WIDE dataset

[0137]

[0138] Table 2 mAP values ​​of various comparison methods on the NUS-WIDE dataset

[0139]

[0140]

[0141] It can be seen from Table 1 and Table 2 that compared with other methods, the method of the present invention has higher retrieval accuracy, whether it is image modality retrieval of text modality or text modality retrieval of image modality.

[0142] The above description is only a preferred embodiment of the present invention and is not intended to be a further limitation of the present invention. All equivalent changes made using the contents of the present specification and drawings are within the protection scope of the present invention.

Claims

1. A multimodal semantic fusion hashing method based on self-paced learning and Fisher criterion, characterized in that: The following steps are involved: Step 1: Obtain large-scale multi-source heterogeneous data, and use a deep network model to extract sample feature information of multimodal data; use a Gaussian mapping function to perform kernelization processing on the sample feature information to obtain the kernelized multimodal data features; Step 2: Using the self-paced learning strategy and the Fisher discriminant criterion to build a multimodal semantic fusion hash learning model, and using the alternating iterative optimization solution method to solve the optimal solution of the multimodal semantic fusion hash learning model parameters; Step 3: Using the optimal solution obtained in step 2, the enhanced projection function learning model is integrated to learn the projection function of projecting multimodal data features to hash codes; Step 4: Based on the multimodal data projection function, perform semantic fusion hash coding processing on the newly arrived multimodal data to generate corresponding semantic fusion hash codes.

2. The multimodal semantic fusion hash learning method based on self-paced learning and Fisher criterion according to claim 1 is characterized in that: Step 1 is as follows: Step 1-1: Obtain large-scale multi-source heterogeneous data, including image data, text data, and label data, and use the deep network model to extract the image sample feature information and text sample feature information of the multimodal data respectively. is the data matrix of the t-th modality, t = 1 and t = 2, X1 represents the image modality data matrix, X2 represents the text modality data matrix, d t is the feature dimension of the t-th mode, n represents the number of samples; Q1∈R d×r is the mapping matrix of the image modality, Q2∈R d×r is the mapping matrix of the text modality, where d is the feature dimension and r is the potential feature matrix dimension; the potential feature matrix of the image modality is represented by V1∈R r×n , the latent feature matrix of the text modality is represented as V2∈R r×n ; The category label matrix Y∈{y1,y2,…y n } has binary restrictions; is the similarity matrix, whose value is in ||.||2 represents the two-norm, ||.|| F represents the F-norm; Step 1-2: Use radial basis function RBF to preprocess image sample data and text sample data, where δ(.) is a Gaussian kernel projection function, and then obtain the nonlinear feature matrix δ(X1) of the image sample data and the nonlinear feature matrix δ(X2) of the text sample data.

3. The multimodal semantic fusion hash learning method based on self-paced learning and Fisher criterion according to claim 2 is characterized in that: Step 2 is as follows: Step 2-1: Construct a self-learning weighted multi-source heterogeneous sample decomposition model: Where Z1 is the weighted diagonal matrix of image samples, is the element value on the diagonal of the i-th row of Z1; Z2 is the weighted diagonal matrix of the text sample, is the element value on the diagonal of the i-th row of Z2; diag(.) is the diagonal matrix representation, and the parameter η controls the order in which the image and text samples are introduced; U is the instance weight matrix, is the value of the element on the diagonal of the i-th row of U; Step 2-2: The self-paced learning weighted multi-source heterogeneous sample decomposition model strategy is further coordinated with the Fisher discriminant criterion to construct a global optimization function corresponding to the multimodal semantic fusion hash learning model: Where Β is the hash code matrix, D1 is the image mapping matrix from the label semantics projected to the image latent feature space, D2 is the text mapping matrix from the label semantics projected to the text latent feature space, W is the non-negative relaxation matrix; C is a predefined constant matrix whose elements are determined by the elements in the category label matrix Y. When Y ij =1, C ij = +1, when Y ij = -1, C ij =-1; T is the label relaxation matrix, T i represents the label relaxation matrix element of the i-th class, is the average value of all samples in the i-th class; M i is the sum of all n in V i The same average column vector; Ω(D1,D2,T) represents the regularization of matrices D1, D2 and T, μ is the balance parameter between label semantics and latent feature space, β is the balance parameter between image latent feature space and text latent feature space, θ is the semantic shift parameter, σ is the Fisher criterion parameter, γ is the regularization parameter, and ⊙ is the Hadamard product of the matrix; Step 2-3: Use the alternating iterative optimization solution method to obtain the optimal solution for the parameters of the multimodal semantic fusion hash learning model.

4. The multimodal semantic fusion hash learning method of self-paced learning and Fisher criterion according to claim 3 is characterized in that: In step 2-3, an alternating iterative optimization solution method is used, that is, when solving a closed-form solution for a parameter, the parameter is regarded as a variable and other variables except the parameter are fixed, specifically: Step 2-3-1, when updating the optimization parameter Q1, fix the other parameters except parameter Q1, and solve the closed-form solution of Q1 as: Q1=(Z1 T Z1δ(X1)UU T V1 T )(V1UU T V1 T Z1 T Z1) -1 ; Step 2-3-2, when updating the optimization parameter Q2, fix the other parameters except parameter Q2, and solve the closed-form solution of Q2 as: <h2 style=";text-align:left;direction:ltr">Q2=(Z2<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> Z2δ(X1)UU<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> V2<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> (V2UU)<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> V2<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> Z2<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> Z2)<h2 style=";text-align:left;direction:ltr"> -1 <h2 style=";text-align:left;direction:ltr"> ; Step 2-3-3: When updating the optimization parameter R, fix all parameters except parameter R and adjust the matrix Perform singular value decomposition to obtain the left orthogonal matrix A1 and the right orthogonal matrix A2, and then solve the closed-form solution of R as follows: Step 2-3-4: When updating the optimization parameter D1, fix the other parameters except parameter D1, and solve the closed-form solution of D1 as follows: Step 2-3-5, when updating the optimization parameter D2, fix the other parameters except parameter D2, and solve the closed-form solution of D2 as: Step 2-3-6, when updating the optimization parameter T, fix the other parameters except parameter T, and solve the closed-form solution of T as follows: Step 2-3-7, when updating the optimization parameter W, fix the other parameters except parameter W, and solve the closed-form solution of W as: W=max(C⊙(TY),0); Step 2-3-8, when updating the optimization parameter V1, other parameters except parameter V1 are fixed, and the global optimization function corresponding to the multimodal semantic fusion hash learning model is converted to: In order to facilitate the explanation of the optimal closed-form solution process for V1, Γ1 is called the intermediate matrix transfer variable of the image, and J1 is the characteristic decomposition transfer variable of the image. For the matrix Γ1J1Γ1 T Perform eigendecomposition to obtain the eigendecomposition left matrix E1 and the eigendecomposition right matrix F1. is the random orthogonal matrix corresponding to the right matrix of the eigendecomposition, is the random orthogonal matrix corresponding to the left matrix of the eigendecomposition, then the optimal closed-form solution of V1 is Step 2-3-9, when updating the optimization parameter V2, other parameters except parameter V2 are fixed, and the global optimization function corresponding to the multimodal semantic fusion hash learning model is converted to: In order to facilitate the explanation of the optimal closed-form solution process for V2, Γ2 is called the intermediate matrix transfer variable of the text, and J2 is the eigendecomposition transfer variable of the text. For the matrix Γ2J2Γ2 T Perform eigendecomposition to obtain the eigendecomposition left matrix E2 and the eigendecomposition right matrix F2. is the random orthogonal matrix corresponding to the right matrix of the eigendecomposition, is the random orthogonal matrix corresponding to the left matrix of the eigendecomposition, then the optimal closed-form solution of V2 is Step 2-3-10, for the sake of convenience, t = 1 and t = 2; then when updating the optimization parameter Z t When the fixed parameter Z is removed t Other parameters except Z t The closed-form solution is: In the formula, Represents the quantization error of the i-th feature, and assigns weights to each feature based on this error That is, each feature will have a corresponding weight value; Step 2-3-11, when updating the optimization parameter U, fix the other parameters except parameter U, and solve the closed-form solution of parameter U as follows: In the formula, It represents the loss function of the i-th instance, and then solves the weights of all instances Step 2-3-12, when updating the optimization parameter B, fix the other parameters except parameter B, and solve the closed-form solution of B as follows: In the formula, sign(.) is the sign function, and its value is 1 when the input parameter value is positive; Otherwise, the value is -1.

5. The multimodal semantic fusion hash learning method of self-paced learning and Fisher criterion according to claim 4 is characterized in that: Step 3 is as follows: Adopt multimodal semantic fusion projection function learning model: In the formula, μ is the balance parameter of the projection function, γ is the weight parameter, and H t is the projection function of the t-th mode, and the formula is optimized to solve the projection function H of multimodal data features projected to the hash code t The optimal closed-form solution of is: H t =(BB T +μI) -1 (lBL T Lδ(X t ) T +ηBδ(X t ) T )(δ(X t )δ(X t ) T +γI) -1 。 6. The multimodal semantic fusion hash learning method of self-paced learning and Fisher criterion according to claim 5 is characterized in that: Step 4 is as follows: Use the multimodal data projection function H trained and learned in step 3 t , perform RBF preprocessing on the newly arrived multimodal data to obtain Gaussian kernel projection features Perform semantic fusion hash coding processing; solve the semantic fusion hash coding B o The optimal closed-form solution is:

Images