Online multi-modal similarity retrieval method and system based on semantic enhancement

Abstract

The invention belongs to the field of multi-modal information retrieval, and discloses an online multi-modal similarity retrieval method and system based on semantic enhancement, and the method comprises the steps: equally grouping an obtained multi-modal flow data set to obtain a plurality of batches of multimedia data blocks, and simulating an online multimedia data flow; constructing a global target optimization function, performing hash code learning storage, and learning a hash function of the multimedia stream data projected to the hash code; and resolving the query sample by using the hash function to obtain a query binary code, and performing Hamming distance resolving on the query binary code and the hash code in the data set to obtain a multi-modal retrieval result corresponding to the to-be-retrieved sample. When a large amount of streaming media data is dynamically fused into a database, the method can effectively reduce the model learning time complexity and improve the Hash code discrimination capability, and can effectively solve the discrete binary constraint problem and retain more semantic information in the Hash code learning process. And rapid and accurate online multi-modal semantic similarity retrieval is realized.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal information retrieval, and specifically relates to an online multimodal similarity retrieval method and system based on semantic enhancement. Background Art

[0002] With the rapid popularization of portable electronic devices and sensors, the data generated is not only large in volume, but also has the characteristics of "complex structure and diverse forms". For example: a web page contains both text information and image information; a video contains both visual information and audio information; a sensor can receive information from different channels such as ultrasound and radar; these information have different sources or expressions, which are called modalities. Under the same medium, there are often multiple different modalities, and data composed of different modalities is called multimodal data. Compared with single-modal information, multimodal data carries richer high-level semantics, which helps to understand things more comprehensively and directly; but multimedia stream data has the characteristics of complex structure, high dimensionality and massive data, and is distributed in different feature spaces. There are problems such as semantic gap and heterogeneous gap, which increases the difficulty of multimodal similarity retrieval. Massive and large-scale multimodal data emerge on the Internet, which is very easy to form logical islands at the semantic level, facing a series of severe challenges such as high information noise, poor quality, low credibility, and semantic ambiguity. Therefore, multimedia information retrieval has become one of the hot issues in current research.

[0003] Hash technology is a technology used in fields such as computer vision and information retrieval to achieve efficient retrieval due to its low storage consumption and low computational cost. Existing hash retrieval technologies are mainly divided into two categories: data-independent hashing and data-dependent hashing. Multimodal hashing methods aim to integrate different feature distributions and dimensions of different modal data to obtain more discriminative binary hash codes. Given that multimedia data usually flows in as data streams in daily life, the demand for online cross-modal hashing learning methods is increasing day by day. Most retrieval methods achieve batch search through continuous training and learning of fixed data sets. With the explosion of the big data era, network data is constantly updated in the form of streams. Existing hashing methods can no longer handle the arrival of new data. How to establish efficient and robust multimodal retrieval methods and systems for large-scale heterogeneous streaming data has become the focus of academic and industrial attention. Summary of the invention

[0004] When a large amount of streaming media data is dynamically integrated into the database, in order to reduce the time complexity of model learning and improve the discrimination ability of hash codes, and at the same time to effectively solve the discrete binary constraint problem and retain more semantic information in the hash code learning process, the present invention proposes an online multimodal similarity retrieval method and system based on semantic enhancement.

[0005] The online multimodal similarity retrieval method based on semantic enhancement of the present invention comprises the following steps:

[0006] Step 1: obtain a multimodal stream data set, and equally group the multimodal stream data set to obtain a total of n batches of multimedia data blocks, construct a sample set corresponding to the multimodal stream data set, and use the multimedia data blocks of each batch to simulate an online multimedia data stream;

[0007] Step 2: When the a-th batch of multimedia data blocks flows into the multimodal streaming data set online, a=2, ...n, the hash code learned in the a-1 batch in the multimodal streaming data set is kept unchanged, and the global objective optimization function is constructed using the a-th batch of data and the a-1 batch of accumulated data to learn and save the hash code;

[0008] Step 3: Using the stream data integrated in the ath batch and the hash code learned in the ath batch, learn a hash function that projects the multimedia stream data to the hash code;

[0009] Step 4: Use a hash function to solve the query sample to obtain a query binary code, and perform Hamming distance calculation on the query binary code and the hash code in the multimodal stream data set to obtain a multimodal retrieval result corresponding to the sample to be retrieved.

[0010] Furthermore, step 1 is specifically as follows:

[0011] Step 1-1: Obtain a multimodal streaming dataset Among them, X m is the feature matrix of the stream data of the mth mode, Y is the label matrix, d m is the feature dimension corresponding to the mth mode, N is the size of the streaming dataset, and c is the number of semantic label types;

[0012] Step 1-2: divide the multimodal stream data set equally, and each group contains the same number of samples of multimodal stream data blocks, so that a total of n batches of multimodal stream data blocks are obtained; the ath batch of multimodal stream data blocks is used Indicates that, n t is the size of the a-th batch of multimodal stream data blocks; The old dataset that already existed when the ath batch of datasets flowed into the dataset; is the size of the dataset when the ath batch flows into the dataset; n i is the size of the i-th batch of data blocks, is the label matrix corresponding to the a-th batch of stream data blocks; When it is the ath batch, there is already a label matrix in the dataset.

[0013] Furthermore, step 2 is specifically as follows:

[0014] Step 2-1: When the a-th batch of multimodal stream data blocks reaches the sample set, the correlation between the multimodal stream data and the hash code is mined to build a matrix three-factor decomposition model of the multimodal stream data:

[0015]

[0016] In the formula, α and β are the trade-off parameters of the model, γ is the penalty parameter of the regularization term, is the transformation matrix between the common subspace of the ath batch and the features of each modal data, is the transformation matrix between the hash code of the ath batch and the common subspace of each modality mapping; represents the F-normal form of the matrix; r is the length of the hash code; and are the hash codes to be learned in batch a and the hash codes already existing in the database; and are the potential common subspace of the ath batch and the existing potential common subspace respectively;

[0017] Step 2-2: When the a-th batch of online data flows in dynamically, an asymmetric semantic label embedding loss function based on the similarity matrix is ​​constructed to mine the semantic relevance between existing data and new data:

[0018]

[0019] In the formula, is the new column-normalized label matrix corresponding to the a-th batch flowing into the dataset, When batch a flows into the dataset, the old column-normalized label matrix in the dataset. For the matrix The i-th column of For the matrix The value of the i-th column of Solution The value of Similarly, For the matrix The i-th column of For the matrix The value of the i-th column of Solution The value of

[0020] In order to reduce the quantization loss of the latent subspace, the orthogonal and balanced constraints are added: and Where I is the identity matrix;

[0021] Step 2-3: Construct a loss function based on the semantic label dragging strategy to fully explore the attribution classification of multi-label semantics:

[0022]

[0023] Where ⊙ is the Hadamard product operator, is the non-negative relaxation matrix to be solved for the ath batch, is the non-negative relaxation matrix that has been solved in the a-1th batch; is the mapping matrix of semantic labels; Drag constant matrix for the new guided semantic label of the current batch, Drag constant matrix for old guide semantic labels; when matrix When the value of the jth row and kth column of is 1, then the matrix The value of the j-th row and k-th column of is +1, otherwise the value is -1; when the matrix When the value of the jth row and kth column of is 1, then the matrix The value of the j-th row and k-th column is +1, otherwise the value is -1.

[0024] Step 2-4: Combine the above steps 2-1 to 2-3 to construct a global objective optimization function of the semantically enhanced multimodal data stream hash learning method:

[0025]

[0026] Where μ and ω are parameters that weigh the impact of semantic embedding and label migration strategies on the overall retrieval performance;

[0027] Step 2-5: Use the alternating iterative optimization solution method to solve the global objective optimization function of step 2-4 to obtain the optimal value of the parameter to be solved.

[0028] Furthermore, steps 2-5 are specifically as follows:

[0029] Step 2-5-1. Update parameters Fixed other parameters, in order to facilitate the description of the parameter solution process, the present invention uses Expressed as a global objective optimization function about The global objective optimization function is then transformed into The optimized subfunction

[0030]

[0031] In the formula, tr(.) represents the trace of the matrix; for Parameters The partial derivative of And set it to zero, get the parameter The closed-form solution of

[0032]

[0033] In the formula, and is the intermediate variable, and is the value solved in the a-1th batch, and and It is solved in this round, and its value is only related to the flow data of batch a;

[0034] Step 2-5-2, Update parameters Fixed other parameters, in order to facilitate the description of the parameter solution process, the present invention uses Expressed as a global objective optimization function about The optimization subfunction of , then the global optimization function

[0035] Turn to About The optimized subfunction

[0036]

[0037] make for Parameters The partial derivative of And set it to zero, get the parameter The closed-form solution of

[0038]

[0039] In the formula, is the intermediate variable, is the value solved for the a-1th batch, and It is solved in this batch, and its value is only related to the flow data of batch a;

[0040] Step 2-5-3, Update parameter G (t) , fix other parameters, in order to facilitate the description of the parameter solution process, the present invention uses Expressed as the global objective optimization function with respect to G (t) The global optimization function is transformed into the optimization subfunction about G (t) The optimized subfunction

[0041]

[0042] Solve For parameter G (t)The partial derivative of and set it to zero, we get the parameter G (t) The closed-form solution of

[0043]

[0044] In the formula, is an intermediate variable; in order to facilitate the solution, the matrix express Using the matrix express but and is the value solved in the a-1th batch, and and The solution is performed in this batch and is only related to the flow data of batch a;

[0045] Step 2-5-4: Update parameters Removal and parameters Irrelevant items, in order to facilitate the description of the parameter solution process, the present invention uses Expressed as a global objective optimization function about The optimization subfunction of , then the global optimization function is transformed into The optimized subfunction

[0046]

[0047] Solve to get parameters The closed-form solution of

[0048]

[0049] Step 2-5-5: Update parameters Removal and parameters Irrelevant items, in order to facilitate the description of the parameter solution process, the present invention uses Expressed as a global objective optimization function about The optimization subfunction of , then the global optimization function is transformed into The optimized subfunction

[0050] In order to solve the optimization subfunction The optimized sub-function The conversion is:

[0051]

[0052] In the formula, is the intermediate variable; let Δ=I-(1 / n t )11 T, where I is the identity matrix and 1 is the identity matrix with all 1s; in order to facilitate the optimization of the subfunction To solve, use the built-in singular value decomposition function svd(.) of matlab to analyze the matrix, that is, use svd(K (t) ΔK (t)T ) Solve to get Σ, U (t) and where Σ is a diagonal matrix of positive eigenvalues, U (t) is the eigenvector corresponding to the positive eigenvalue, Let the matrix Q be the remaining eigenvectors corresponding to zero eigenvalues. (t) =ΔK (t)T U (t) Σ -12 , using Matlab's built-in matrix orthogonal normalized basis function orth(.) to calculate the matrix Q (t) Perform orthogonal operations to obtain the orthogonal basis matrix And the matrix Perform Schmidt orthogonalization to obtain the matrix Then the parameters are solved. The closed-form solution of

[0053]

[0054] Step 2-5-6: Update parameters Removal and parameters Irrelevant terms, the global objective optimization function is transformed into Sub-optimization function:

[0055]

[0056] Solving about parameters The closed-form solution of is:

[0057]

[0058] In the formula, is the intermediate variable, is the value solved for the a-1th batch, and It is solved in this batch, and its value is only related to the flow data of batch a.

[0059] Furthermore, step 3 is specifically as follows:

[0060] Step 3-1: Learned from step 2 Construct the loss function corresponding to the hash function learning model:

[0061]

[0062] Where ⊙ represents the Hadamard product operator, represents the hash function of the online streaming data to be learned, is the offset direction matrix of the ath batch and the values ​​in the matrix are non-negative. is the offset direction matrix obtained by solving the a-1th batch, and ξ is a coefficient used to avoid overfitting;

[0063] Step 3-2: Use the alternating iterative optimization method to obtain the optimal parameters The value of .

[0064] Further, in step 3-2, the optimal parameters are obtained The value of is:

[0065] Step 3-2-1. Update parameters Removal and parameters Irrelevant items, in order to facilitate the description of the parameter solution process, the present invention uses Expressed as a global objective optimization function about The optimization subfunction of , then the global objective optimization function is transformed into Sub-optimization function

[0066]

[0067] Step 3-2-2: Update solution parameters Removal and parameters Irrelevant terms, the optimization function is transformed into Sub-optimization function of ;

[0068]

[0069] Solve for parameters The closed-form solution of is:

[0070]

[0071] Furthermore, step 4 is specifically as follows: using a hash function to solve the query sample to obtain a query binary code, performing an XOR operation on the query binary code and the hash code in the stream database to obtain a Hamming distance; sorting the Hamming distances from small to large, and feeding back the first t data as the multimodal retrieval result corresponding to the sample to be retrieved, where t is a positive integer; specifically as follows:

[0072] The multimodal data to be retrieved is x q , according to the obtained Map the discrete hash code of the modality data to be retrieved:

[0073]

[0074] The present invention also provides an online multimodal similarity retrieval system based on semantic enhancement, which is used to implement the method, and includes a multimodal data stream acquisition module, a hash code learning module, a hash function learning module and a multimodal similarity retrieval module;

[0075] The multimodal data stream acquisition module is used to acquire a multimodal stream data set and process the multimodal data set to obtain n batches of multimodal data blocks;

[0076] The hash code learning module is used to keep the hash code corresponding to the a-1th batch of multimodal data unchanged, and to learn and save the ath batch of hash codes for the ath batch of multimodal data blocks flowing into the heterogeneous database;

[0077] The hash function learning module uses the hash codes learned in the ath batch to learn the hash functions of the corresponding multimodal data;

[0078] The multimodal similarity retrieval module obtains the corresponding query binary code through the learned hash function, performs Hamming distance calculation between the obtained query binary code and the hash code in the data set, and obtains the retrieval result of the modal data to be retrieved.

[0079] The beneficial effects of the present invention are:

[0080] 1) In order to solve the problem of efficient multimodal hash learning in the process of dynamic integration of large-scale streaming data, the present invention mines the correlation between multimodal streaming data and hash codes, designs a semantically enhanced hash learning model, and adopts a two-step hash learning strategy to first learn the hash code and then further learn the hash function of the online streaming data; in the process of hash code learning for each round of data blocks, the hash code learned in the previous round of streaming data is kept unchanged, and there is no need to repeatedly train the old data in the database, so that efficient hash code learning for online streaming data can be performed, so that in the process of multimedia online streaming data learning, more semantic information can be included while achieving a balance between time and space overhead. The present invention not only has scalability and adaptability, but also has good retrieval performance;

[0081] 2) Considering the continuous growth of multimodal data, in order to solve the discrete binary constraint problem involved in the multimodal learning model of streaming data and retain more semantic information in the hash code learning process, the present invention constructs a semantically enhanced online multimodal similarity retrieval model. Compared with other streaming data multimodal similarity retrieval methods, the present invention not only enhances the fine-grained semantic information of the hash code, but also meets the growing demand for online learning, accelerates the retrieval rate between multimodal data, and is conducive to realizing fast similarity retrieval of multimodal large-scale online streaming data. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 is a flow chart of the method of the present invention;

[0083] Figure 2 It is a framework diagram of the method described in the present invention. DETAILED DESCRIPTION

[0084] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0085] like Figure 1 and Figure 2 As shown, the online multimodal similarity retrieval method based on semantic enhancement described in the present invention comprises the following steps:

[0086] Step 1: extract features from image modal samples and text modal samples in a multimodal stream data set respectively, construct a multimodal stream data set using the extracted features, and equally group the multimodal stream data set to obtain a total of n batches of multimedia data blocks, construct a sample set corresponding to a heterogeneous multimedia data stream, and use the multimedia data blocks of each batch to simulate an online multimedia data stream;

[0087] Step 2: When the a-th batch of multimedia data blocks flows into the multimodal data set online, a=2, ...n, the hash code learned in the a-1 batch in the multimodal streaming data set is kept unchanged, and the global objective optimization function is constructed using the a-th batch of data and the a-1 batch of accumulated data to learn and save the hash code;

[0088] Step 3: Using the stream data integrated in the ath batch and the hash code learned in the ath batch, learn a hash function that projects the multimedia stream data to the hash code;

[0089] Step 4: Use a hash function to solve the query sample to obtain a query binary code, perform Hamming distance calculation on the query binary code and the hash code in the stream data set, and obtain a multimodal retrieval result corresponding to the sample to be retrieved.

[0090] Taking image mode and text mode as examples, the present invention is further described in detail.

[0091] The present invention first constructs a convolutional neural network VGG16 model with 13 convolutional layers, 3 fully connected layers and 5 pooling layers, and trains the VGG16 model. Then, the trained VGG16 model is used to extract features of image modality samples in a multimodal streaming dataset.

[0092] Then, a global vector GloVe model is constructed to model the text data. The GloVe model is trained by converting words into vector forms with semantic characteristics. The trained GloVe model is then used to extract features from text modality samples in the multimodal streaming dataset.

[0093] Construct a multimodal streaming dataset based on the features extracted above Among them, X m is the stream data feature matrix of the mth mode, that is, X1 is the stream data feature matrix of the image mode, X2 is the stream data feature matrix of the text mode; Y is the label matrix, d m is the feature dimension corresponding to the mth modality, that is, d1=512 is the feature dimension of the image modality, d2=1386 is the feature dimension of the text modality; N=20000 is the size of the streaming dataset, and c=24 is the number of semantic label types.

[0094] The multimodal streaming data set is equally divided, and the number of samples of streaming data blocks contained in each group is 2000, and 10 groups of multimodal streaming data blocks are obtained. The last group of multimodal streaming data blocks is used as a test set, and the other 9 groups of multimodal streaming data blocks are used as training sets. For the convenience of explanation, the present invention regards 9 groups as 9 batches, that is, n=9 batches of multimedia streaming data blocks; the ath batch of streaming data blocks is used Indicates that the image modality feature in the a-th batch of stream data blocks is The text modality feature is Among them, n t =2000 is the size of the a-th batch of data blocks; When the ath batch of data flows into the dataset, the existing old dataset, that is, the image modality features in the existing old dataset are The text modality features in the existing old dataset are is the size of the dataset when the ath batch flows into the dataset; n i =2000 is the size of the i-th batch of data blocks, is the label matrix corresponding to the a-th batch of stream data blocks; When it is the ath batch, there is already a label matrix in the dataset.

[0095] Step 2 is as follows:

[0096] Step 2-1: When the a-th batch of stream data blocks reaches the sample set, the correlation between the multimodal stream data and the hash code is mined to build a matrix three-factor decomposition model of the multimodal stream data:

[0097]

[0098] In the formula, the present invention sets α=1 and β=0.01 as the trade-off parameters of the model, and γ=0.01 as the penalty parameter of the regularization term. is the transformation matrix between the common subspace of the ath batch and the image data features, is the transformation matrix between the common subspace of the ath round and the text data features.

[0099] is the transformation matrix between the hash code of the ath batch and the common subspace of the image modality mapping, is the transformation matrix between the hash code of the ath round and the common subspace of the text modality mapping; represents the F-normal form of the matrix; r=32 is the length of the hash code; and are the hash codes to be learned in round a and the hash codes already existing in the database; and are the potential common subspace of the ath batch and the existing potential common subspace respectively;

[0100] Step 2-2: When the a-th batch of online data flows in dynamically, an asymmetric semantic label embedding loss function based on the similarity matrix is ​​constructed to mine the semantic relevance between existing data and new data:

[0101]

[0102] In the formula, is the new column-normalized label matrix corresponding to the a-th batch flowing into the dataset, The old column-normalized label matrix in the dataset when batch a flows into the dataset. For the matrix The i-th column of For the matrix The value of the i-th column of Solution The value of Similarly, For the matrix The i-th column of For the matrix The value of the i-th column of Solution The value of

[0103] In order to reduce the quantization loss of the latent subspace, the orthogonal and balanced constraints are added: and Where I is the identity matrix;

[0104] Step 2-3: Construct a loss function based on the semantic label dragging strategy to fully explore the attribution classification of multi-label semantics:

[0105]

[0106] Where ⊙ is the Hadamard product operator, is the non-negative relaxation matrix to be solved for the ath batch, is the non-negative relaxation matrix that has been solved in the a-1th batch; is the mapping matrix of semantic labels; Drag constant matrix for the new guided semantic label of the current batch, Drag the constant matrix for the old guided semantic label; when the matrix Y (t) When the value of the jth row and kth column of is 1, then the matrix The value of the j-th row and k-th column of is +1, otherwise the value is -1; when the matrix When the value of the jth row and kth column of is 1, then the matrix The value of the j-th row and k-th column is +1, otherwise the value is -1.

[0107] Step 2-4: Combine the above steps 2-1 to 2-3 to construct a global objective optimization function of the semantically enhanced multimodal data stream hash learning method:

[0108]

[0109] Where, μ = 0.0001 and ω = 0.001 are parameters that weigh the impact of semantic embedding and label migration strategies on the overall retrieval performance;

[0110] Step 2-5: Use the alternating iterative optimization method to solve the global objective optimization function of step 2-4 to obtain the optimal value of the parameter to be solved; specifically:

[0111] Step 2-5-1, Update parameter W1 (t) , fix other parameters, and get the value of parameter W1 (t) The closed-form solution of

[0112]

[0113] In the formula, and is the value solved in the a-1th batch, and and It is solved in this batch, and its value is only related to the flow data of batch a;

[0114] Step 2-5-2, Update parameters Fix other parameters and get the parameters The closed-form solution of

[0115]

[0116] In the formula, and is the value solved in the a-1th batch, and and It is solved in this batch, and its value is only related to the flow data of batch a;

[0117] Step 2-5-3: Update parameters Fix other parameters and get the parameters The closed-form solution of

[0118]

[0119] In the formula, is the intermediate variable, is the value solved for the a-1th batch, and It is solved in this batch, and its value is only related to the flow data of batch a;

[0120] The same solution is obtained The closed-form solution of

[0121]

[0122] Step 2-5-4, update parameter G (t) , fix other parameters, and get the value of parameter G (t) The closed-form solution of

[0123]

[0124] In the formula, is an intermediate variable; in order to facilitate the solution, the matrix express Using the matrix express but and is the value solved for the a-1th batch, and and The solution is performed in this batch and is only related to the flow data of batch a;

[0125] Step 2-5-5: Update parameters Removal and parameters Irrelevant terms, solve to get parameters The closed-form solution of

[0126]

[0127] Step 2-5-6: Update parameters Removal and parameters Irrelevant items, in order to facilitate the description of the parameter solution process, the present invention uses Expressed as a global objective optimization function about The optimization subfunction of , then the global optimization function is transformed into The optimized subfunction

[0128]

[0129] In order to solve the optimization subfunction The optimized sub-function The conversion is:

[0130]

[0131] In the formula, is the intermediate variable; let Δ=I-(1 / n t )11 T , where I is the identity matrix and 1 is the identity matrix with all 1s; in order to facilitate the optimization of the subfunction To solve, use the built-in singular value decomposition function svd(.) of matlab to analyze the matrix, that is, use svd(K (t) ΔK (t)T ) Solve to get Σ, U (t) and where Σ is a diagonal matrix of positive eigenvalues, U (t) is the eigenvector corresponding to the positive eigenvalue, Let the matrix Q be the remaining eigenvectors corresponding to zero eigenvalues. (t) =ΔK (t)T U (t) Σ -12 , using Matlab's built-in matrix orthogonal normalized basis function orth(.) to calculate the matrix Q (t) Perform orthogonal operations to obtain the orthogonal basis matrix And the matrix Perform Schmidt orthogonalization to obtain the matrix Then the solution is obtained about the parameters The closed-form solution of

[0132]

[0133] Step 2-5-7: Update parameters Removal and parameters Irrelevant terms, solve for parameters The closed-form solution of is:

[0134]

[0135] In the formula, is the intermediate variable, is the value solved for the a-1th batch, and It is solved in this round, and its value is only related to the flow data of batch a.

[0136] Step 3 is as follows:

[0137] Step 3-1: Learned from step 2 Construct the loss function corresponding to the hash function learning model:

[0138]

[0139] Where ⊙ represents the Hadamard product operator, P1 (t) It is represented as the hash function of the online streaming data corresponding to the image modality, P2 (t) It is represented as a hash function of online streaming data corresponding to the text modality; is the offset direction matrix of the image modality in batch a and the values ​​in the matrix are non-negative, is the offset direction matrix of the text mode in batch a and the values ​​in the matrix are non-negative. is the offset direction matrix obtained by solving the image modality in batch a-1, is the offset direction matrix obtained by solving the text mode in the a-1th batch, and ξ is a coefficient used to avoid overfitting;

[0140] Step 3-2: Use the alternating iterative optimization method to obtain the optimal parameter P1 (t) ,P2 (t) , The value of .

[0141] Further, in step 3-2, the optimal parameter P1 is obtained (t) ,P2 (t) , The value of is:

[0142] Step 3-2-1, Update parameter P1 (t) , remove and parameter P1 (t) Irrelevant terms, solve for parameter P1 (t) The closed-form solution of is:

[0143]

[0144] Step 3-2-2: Similarly, solve the parameter P2 (t) The closed-form solution of is:

[0145]

[0146] Step 3-2-3: Update solution parameters Removal and parameters Irrelevant terms, solve for parameters The closed-form solution of is:

[0147]

[0148] Step 3-2-4: Similarly, solve the parameters The closed-form solution of is:

[0149]

[0150] In step 4, the query sample is solved using a hash function to obtain a query binary code, and the query binary code is XORed with the hash code in the stream database to obtain the Hamming distance; the Hamming distances are sorted from small to large, and the first t data are fed back as the multimodal retrieval results corresponding to the sample to be retrieved, where t is a positive integer; specifically:

[0151] The multimodal data to be retrieved is image modality sample x q , according to the obtained P1 (t) Map the discrete hash code of the modality data to be retrieved:

[0152] B q =sgn(P1 (t) x q ).

[0153] The present invention also provides an online multimodal similarity retrieval system based on semantic enhancement, which is used to implement the method, and includes a multimodal data stream acquisition module, a hash code learning module, a hash function learning module and a multimodal similarity retrieval module;

[0154] The multimodal data stream acquisition module is used to acquire a multimodal stream data set and process the multimodal data set to obtain n batches of multimodal data blocks;

[0155] The hash code learning module is used to keep the hash code corresponding to the a-1th batch of multimodal data unchanged, and to learn and save the ath batch of hash codes for the ath batch of multimodal data blocks flowing into the heterogeneous database;

[0156] The hash function learning module uses the hash codes learned in the ath batch to learn the hash functions of the corresponding multimodal data;

[0157] The multimodal similarity retrieval module obtains the corresponding query binary code through the learned hash function, performs Hamming distance calculation between the obtained query binary code and the hash code in the data set, and obtains the retrieval result of the modal data to be retrieved.

[0158] The above description is only a preferred embodiment of the present invention and is not intended to be a further limitation of the present invention. All equivalent changes made using the contents of the present specification and drawings are within the protection scope of the present invention.

Claims

1. An online multimodal similarity retrieval method based on semantic enhancement, characterized by: The following steps are involved: Step 1: Obtain a multimodal stream dataset and equally group the multimodal stream dataset to obtain n batches of multimedia data blocks, construct a sample set corresponding to the multimodal stream dataset, and use the multimedia data blocks of each batch to simulate an online multimedia data stream; Step 2: When the ath batch of multimedia data blocks flows online into the multimodal streaming dataset, a = 2, ... n, the hash code learned in the a-1th batch of the multimodal streaming dataset remains unchanged, and the global objective optimization function is constructed using the ath batch of data and the a-1th batch of accumulated data to learn and save the hash code; Step 3: Using the stream data incorporated in the ath batch and the hash code learned in the ath batch, learn a hash function that projects the multimedia stream data into the hash code; Step 4: Use a hash function to solve the query sample to obtain a query binary code, and perform Hamming distance calculation on the query binary code and the hash code in the multimodal stream data set to obtain a multimodal retrieval result corresponding to the sample to be retrieved.

2. The online multimodal similarity retrieval method based on semantic enhancement according to claim 1 is characterized in that: Step 1 is as follows: Step 1-1: Obtain a multimodal streaming dataset Among them, X m is the feature matrix of the stream data of the mth mode, Y is the label matrix, d m is the feature dimension corresponding to the mth modality, N is the size of the streaming dataset, and c is the number of semantic label types; Step 1-2: divide the multimodal stream data set equally. Each group contains the same number of samples of multimodal stream data blocks, and obtain a total of n batches of multimodal stream data blocks; the ath batch of multimodal stream data blocks is used Indicates that, n t is the size of the multimodal stream data block in batch a; The old dataset that already existed when the ath batch of data was imported. is the size of the dataset when the ath batch flows into the dataset; n i is the size of the data block in batch i, is the label matrix corresponding to the a-th batch of stream data blocks; When it is the ath batch, there is already a label matrix in the dataset.

3. The online multimodal similarity retrieval method based on semantic enhancement according to claim 2, characterized in that: Step 2 is as follows: Step 2-1: When the a-th batch of multimodal stream data blocks reaches the sample set, the correlation between the multimodal stream data and the hash code is mined to build a matrix three-factor decomposition model of the multimodal stream data: In the formula, α and β are the trade-off parameters of the model, γ is the penalty parameter of the regularization term, is the transformation matrix between the common subspace of the ath round and the features of each modal data, is the transformation matrix between the hash code of the ath batch and the common subspace of each modality mapping; represents the F-normal form of the matrix; r is the length of the hash code; and are the hash codes to be learned in batch a and the hash codes already existing in the database; and are the potential common subspace of batch a and the existing potential common subspace respectively; Step 2-2: When the a-th batch of online data flows in dynamically, an asymmetric semantic label embedding loss function based on the similarity matrix is ​​constructed to mine the semantic relevance between existing data and new data: Where, is the new column-normalized label matrix when the corresponding batch a flows into the dataset, When the ath batch flows into the dataset, the old column-normalized label matrix in the dataset; is a matrix The i-th column of is a matrix The value of the i-th column can be used Solution The value of is a matrix The i-th column of is a matrix The value of the i-th column of Solution The value of In order to reduce the quantization loss of the latent subspace, the orthogonality and balance constraints are added: and Where I is the identity matrix; Step 2-3: Construct a loss function based on the semantic label dragging strategy to fully explore the attribution classification of multi-label semantics: Where ⊙ is the Hadamard product operator, is the non-negative relaxation matrix to be solved for the ath batch, is the non-negative relaxation matrix that has been solved in the a-1th batch; is the mapping matrix of semantic labels; Drag a constant matrix for the new guided semantic labels of the current batch, Drag constant matrix for old guide semantic labels; when matrix When the value of the jth row and kth column of is 1, then the matrix The value of the jth row and kth column of is +1, otherwise the value is -1; when the matrix When the value of the jth row and kth column of is 1, then the matrix The value of row j and column k is +1, otherwise the value is -1; Step 2-4: Combine steps 2-1 to 2-3 above to construct a global objective optimization function for the semantically enhanced multimodal data stream hash learning method: Where μ and ω are parameters that weigh the impact of semantic embedding and label transfer strategies on the overall retrieval performance; Step 2-5: Use the alternating iterative optimization method to solve the global objective optimization function of step 2-4 to obtain the optimal value of the parameter to be solved.

4. The online multimodal similarity retrieval method based on semantic enhancement according to claim 3 is characterized in that: Steps 2-5 are as follows: Step 2-5-1. Update parameters Fixed other parameters, in order to facilitate the description of the parameter solution process, use Expressed as a global objective optimization function about The optimization subfunction of , then the global objective optimization function is transformed into The optimized subfunction Where tr(.) represents the trace of the matrix; for Parameters The partial derivative of And set it to zero, get the parameter The closed-form solution of Where, and is the intermediate variable, and is the value solved for the a-1th batch, and and It is solved in this round, and its value is only related to the flow data of batch a; Step 2-5-2, Update parameters Fixed other parameters, in order to facilitate the description of the parameter solution process, use Expressed as a global objective optimization function about The optimization subfunction of , then the global optimization function is transformed into The optimized subfunction make for Parameters The partial derivative of And set it to zero, get the parameter The closed-form solution of Where, is the intermediate variable, is the value solved for the a-1th batch, and It is solved in this batch, and its value is only related to the flow data of batch a; Step 2-5-3, update parameter G (t) , fix other parameters, and use Expressed as the global objective optimization function about G (t) The optimization subfunction of G is converted into the global optimization function (t) The optimized subfunction Solve For parameter G (t) The partial derivative of G is set to zero, and the (t) The closed-form solution of Where, is an intermediate variable; in order to facilitate the solution, the matrix express Using the matrix express but and is the value solved for the a-1th batch, and and The solution is performed in this batch and is only related to the flow data of batch a; Step 2-5-4, Update parameters Removal and parameters In order to describe the parameter solution process, we use Expressed as a global objective optimization function about The optimization subfunction of , then the global optimization function is transformed into The optimized subfunction Solve to get parameters The closed-form solution of Step 2-5-5, Update parameters Removal and parameters In order to describe the parameter solution process, we use Expressed as a global objective optimization function about The optimization subfunction of , then the global optimization function is transformed into The optimized subfunction In order to solve the optimization subfunction Optimize the subfunction Convert it to: Where, is the intermediate variable; let Δ=I-(1 / n t )11 T , where I is the identity matrix and 1 is the identity matrix with all 1s; in order to facilitate the optimization of the subfunction To solve, use the built-in singular value decomposition function svd(.) of matlab to analyze the matrix, that is, use svd(K (t) ΔK (t)T ) Solve to get Σ, U (t) and Where Σ is a diagonal matrix of positive eigenvalues, U (t) is the eigenvector corresponding to the positive eigenvalue, are the remaining eigenvectors corresponding to zero eigenvalues; let the matrix Q (t) =ΔK (t)T U (t) Σ -12 , using Matlab's built-in matrix orthogonal normalized basis function orth(.) to calculate the matrix Q (t) Perform orthogonal operations to obtain the orthogonal basis matrix And the matrix Perform Schmidt orthogonalization to obtain the matrix Then the parameters are solved and obtained The closed-form solution of Steps 2-5-6: Update parameters Removal and parameters Irrelevant terms, the global objective optimization function is transformed into Sub-optimization function of: Solving about parameters The closed-form solution of : Where, is the intermediate variable, is the value solved for the a-1th batch, and It is solved in this batch, and its value is only related to the flow data of batch a.

5. The online multimodal similarity retrieval method based on semantic enhancement according to claim 4 is characterized in that: Step 3 is as follows: Step 3-1: Learned from step 2 Construct the loss function corresponding to the hash function learning model: Where ⊙ represents the Hadamard product operator, represents the hash function of the online streaming data to be learned, is the offset direction matrix of batch a and the values ​​in the matrix are non-negative, is the offset direction matrix obtained by solving the a-1th batch, and ξ is a coefficient used to avoid overfitting; Step 3-2: Use the alternating iterative optimization method to obtain the optimal parameters The value of .

6. The online multimodal similarity retrieval method based on semantic enhancement according to claim 5 is characterized in that: In step 3-2, get the optimal parameters The value of is: Step 3-2-1. Update parameters Removal and parameters In order to describe the parameter solution process, we use Expressed as a global objective optimization function about The optimization subfunction of , then the global objective optimization function is transformed into Sub-optimization function Solve Parameters The partial derivative of is set to zero, and the solution is obtained about the parameter The closed-form solution of : Step 3-2-2, update solution parameters Removal and parameters Irrelevant terms, the optimization function is transformed into Sub-optimization function of ; Solve for parameters The closed-form solution of :

7. The online multimodal similarity retrieval method based on semantic enhancement according to claim 6 is characterized in that: Step 4 is as follows: Use the hash function to solve the query sample to obtain the query binary code, perform an XOR operation on the query binary code and the hash code in the stream database to obtain the Hamming distance; sort the Hamming distances from small to large, and feed back the first t data as the multimodal retrieval result corresponding to the sample to be retrieved, where t is a positive integer; specifically: The multimodal data to be retrieved is x q , according to the obtained Map the discrete hash code of the modality data to be retrieved:

8. An online multimodal similarity retrieval system based on semantic enhancement, characterized by: The system is used to implement the method according to any one of claims 1 to 7, and the system comprises: a multimodal data stream acquisition module, a hash code learning module, a hash function learning module, and a multimodal similarity retrieval module; The multimodal data stream acquisition module is used to obtain a multimodal stream data set and process the multimodal data set to obtain n batches of multimodal data blocks; The hash code learning module is used to keep the hash code corresponding to the a-1th batch of multimodal data unchanged, and learn and save the ath batch of hash codes for the ath batch of multimodal data blocks flowing into the heterogeneous database; The hash function learning module uses the hash codes learned in the ath batch to learn the hash functions of the corresponding multimodal data; The multimodal similarity retrieval module obtains the corresponding query binary code through the learned hash function, and performs Hamming distance calculation between the obtained query binary code and the hash code in the data set to obtain the retrieval result of the modal data to be retrieved.

Images