A cross-modal hash retrieval method based on a dual variational autoencoder

Abstract

This invention discloses a cross-modal hash retrieval method based on a dual variational autoencoder, comprising: inputting image and text information of samples into a preset feature extraction network to obtain original image and text features; constructing an image variational autoencoder model and a text variational autoencoder model based on the features in the image and text modalities, embedding the original image and text features into a common latent space; and constructing a loss function to provide constraints for the hash learning of the feature extraction network. The improvement in training method in this invention can effectively utilize nearest neighbor information in sparse graphs, which can not only reduce the performance loss caused by not utilizing global information, but also improve the performance of highly relevant retrieval.

Description

Technical Field

[0001] This invention relates to the field of computer technology. More specifically, it relates to a cross-modal hash retrieval method based on a dual variational autoencoder. Background Technology

[0002] The past few decades have witnessed the rapid growth of diverse content types on the internet. The same event or object can be described as different kinds of data, which can be termed multimodal data with heterogeneous attributes. This vast amount of multimodal data influences people's information needs and how they search for information on the internet. Cross-modal retrieval aims to use query data to search for relevant data in other different modalities. For example, using descriptive text to retrieve relevant images from a database.

[0003] Cross-modal retrieval has attracted increasing research attention. The most challenging aspect of cross-modal retrieval is measuring the similarity between features of different modalities in the data, a phenomenon known as heterogeneity gap. To support similarity relationship search, it is necessary to map incomparable data to comparable features.

[0004] To date, unsupervised hashing methods can be broadly categorized into two types based on whether they utilize pairwise information (i.e., correspondences provided before the training phase). The first type embeds high-level features into a latent Hamming space and constrains quantization and distribution losses. While this algorithm has achieved some impressive results, its performance is far from satisfactory. To not only increase the variance of the hash codes but also collect more original information, the second type focuses on preserving pairwise information to construct similarity constraints. Some of these methods preserve similarity information through graph structures. Despite these breakthroughs, the task still faces two major problems. First, dense graphs containing predefined local neighborhood information, predominantly present in mini-batch deep learning models, result in a lot of redundant information. This means that most of the proximity information is useless and can mislead neighborhood relationships in the common Hamming space, leading to the learning of redundant hash codes. Second, previous methods failed to model the posterior distribution of the observed data, relying solely on similarity information during training.

[0005] In view of these problems, we propose a cross-modal hash retrieval method based on a dual variational autoencoder. Summary of the Invention

[0006] One object of the present invention is to provide a cross-modal hash retrieval method based on a dual variational autoencoder to solve at least one of the problems existing in the prior art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A cross-modal hash retrieval method based on a dual variational autoencoder, the method comprising:

[0009] The image modal information and text modal information of the sample are input into a preset feature extraction network to obtain the original image features and text features;

[0010] Based on features in the image modality and features in the text modality, we construct image variational autoencoder models and text variational autoencoder models, embedding the original image features and original text features into a common latent space;

[0011] A loss function is constructed to provide constraints for the hash learning of the feature extraction network.

[0012] Optionally, the preset feature extraction network includes: an image feature extraction network consisting of the first 7 fully connected layers of a VGG16 network, and a text feature extraction network consisting of a general sentence encoder.

[0013] Optionally, the image variational autoencoder and the text variational autoencoder include an encoder and a decoder; the encoder and decoder include three layers, with a Leaky-Rule layer and a Batch-Norm layer inserted between two layers.

[0014] Optionally, the constructed loss function includes an intra-modal reconstruction loss function, an inter-modal reconstruction loss function, a clustering loss function, a modal alignment loss, and a reconstruction consistency loss function.

[0015] Optionally, the intra-modal reconstruction loss function for

[0016]

[0017] Where x I and z I These represent the original feature vector in the image modality and the continuous latent space feature vector obtained after the image autoencoder, respectively. The continuous latent space feature distribution obtained from the original feature distribution in the representative image modality. The approximate distribution of z I =μ I +σ I ⊙∈, where μ I and σ I The vectors obtained through the encoder in the image autoencoder represent the mean vector and standard deviation vector of the multivariate normal distribution, respectively. The encoder parameter is φ I , The prior distribution in the continuous latent space of the image modality is also assumed to be a normal distribution. Kullback-Leibler (KL) divergence represents the ratio between the prior distribution and the approximate distribution of the posterior distribution of latent space features in an image modality. This indicates the expected value, and log(·) represents the logarithmic operation with the natural constant as the base. This represents the original feature distribution obtained from the continuous latent space feature distribution in the image modality. The experiment uses a decoder from an image autoencoder, with the decoder's parameters being θ. I , It represents the lower bound of the maximum likelihood estimate of the original feature distribution reconstructed under the image modality;

[0018] x T and z T These represent the original feature vector in the text modality and the continuous latent space feature vector obtained after the image autoencoder, respectively. The continuous latent space feature distribution obtained from the original feature distribution in the text modality represents the continuous latent space feature distribution. The approximate distribution of z T =μ T +σ T ⊙∈, where μ T and σ T The vectors obtained through the encoder in the text autoencoder represent the mean vector and standard deviation vector of the multivariate normal distribution, respectively. The encoder parameter is φ T , The prior distribution in the continuous latent space of the text modality is also assumed to be a normal distribution. Kullback-Leibler (KL) divergence represents the distance between the prior distribution and the approximate distribution of the posterior distribution of latent space features in the text modality. This represents the original feature distribution obtained from the continuous latent space feature distribution in the text modality. The experiment uses a decoder from an image autoencoder, with the decoder parameter being φ. T , It represents the lower bound of the maximum likelihood estimate of the original feature distribution reconstructed under the text modality.

[0019] Optionally, the inter-modal reconstruction loss function The format is as follows:

[0020]

[0021]

[0022] Optionally, the clustering loss function includes

[0023] Modal clustering loss function

[0024]

[0025] in, This represents the probability that the i-th sample is assigned to the j-th cluster center in the image modality. This represents the probability that the i-th sample belongs to the j-th cluster center in the image modality; This represents the probability that the i-th sample is assigned to the j-th cluster center in the text modality; This represents the probability that the i-th sample belongs to the j-th cluster center in the text modality.

[0026] Optionally, the modal alignment loss The format is as follows:

[0027]

[0028] Among them ||·|| F Let ||·||2 be the Frobenius norm of the matrix, and ||·||2 be the 2-norm of the matrix. and Let and represent the mean vector and standard deviation vector of the multivariate normal distribution generated by the encoder under image modality, respectively. and These represent the mean vector and standard deviation vector of the multivariate normal distribution generated by the encoder in the text modality for the i-th sample, respectively.

[0029] Optionally, the reconstruction consistency loss function The format is as follows:

[0030]

[0031] in

[0032]

[0033] ||·|| represents the magnitude of the vector, and (·) T Let be the transpose of a matrix or vector, cos(·,·) be the cosine similarity between vectors, and tanh(·) be the tangent activation function; λ and γ are preset weight parameters, n is the preset number of training samples per run, and β is a clustering density hyperparameter. β > 1 indicates that the similarity score of hash code pairs needs to be greater than 1, which correspondingly makes the nodes in the Hamming space dense, while β < 1 indicates that the similarity score of hash code pairs needs to be less than 1, which correspondingly makes the nodes in the Hamming space sparse. As the number of training steps increases, κ tends to infinity, which can make the tangent activation function gradually become the sign function. Let and The elements in the array gradually move towards +1 and -1. and It is approximately a binary vector.

[0034] Optionally, the loss function The format is as follows:

[0035]

[0036] Where ∈1, ∈2, ∈3 and ∈4 are hyperparameters of the balance loss.

[0037] The beneficial effects of this invention are as follows:

[0038] This invention proposes a novel deep learning framework that learns a compact and accurate hash representation of multimodal information through a dual VAE model. While updating cluster centers, it jointly optimizes the latent space embedding features of the features. The improvement in training method of this invention can effectively utilize the nearest neighbor information in sparse graphs, which can not only reduce the performance loss caused by not utilizing global information, but also improve the performance of highly relevant retrieval. Attached Figure Description

[0039] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0040] Figure 1 A flowchart of the method described in this invention is shown.

[0041] Figure 2 A schematic diagram of a specific embodiment of the present invention is shown. Detailed Implementation

[0042] To more clearly illustrate the present invention, the following description is in conjunction with preferred embodiments and accompanying drawings. Figure 1 The present invention will be further described below. Similar components in the accompanying drawings are indicated by the same reference numerals. Those skilled in the art should understand that the following detailed description is illustrative rather than restrictive and should not be construed as limiting the scope of protection of the present invention.

[0043] A cross-modal hash retrieval method based on a dual variational autoencoder is proposed. The method generates distributions of visual and textual modal data on possible values ​​of latent features, respectively. It uses a deep embedding clustering method to aggregate the average value of latent features, while updating the cluster centers and the embedding process. Then, it forces the aggregation of the mean values ​​of multimodal data from the same category to align the distributions of the two modalities, thereby generating the same posterior distribution and thus generating a more accurate hash code.

[0044] The method includes:

[0045] The image and text information of the sample are input into a preset feature extraction network to obtain the original image and text features;

[0046] Based on features from image and text modalities, we construct image variational autoencoder models and text variational autoencoder models, embedding the original image features and original text features into a common latent space;

[0047] A loss function is constructed to provide constraints for the hash learning of the feature extraction network.

[0048] One specific implementation method,

[0049] S1 uses a pre-trained deep neural network to obtain feature representations of images and text;

[0050] S2 constructs image deep hashing networks and text deep hashing networks;

[0051] S3 inputs training data into the image variational autoencoder model and the text variational autoencoder model;

[0052] S4 clusters the sample mean feature vectors output by the image encoder and text encoder and aligns them with the auxiliary distribution;

[0053] The mean feature vector and standard deviation vector of the samples output by the image encoder and text encoder are sequentially modally aligned.

[0054] S5 combines the sample mean feature vector and standard deviation vector output by the image encoder into continuous latent space image features;

[0055] The sample mean feature vector and standard deviation vector output by the text encoder are combined to form continuous latent space text features;

[0056] S6 uses the tangent activation function to generate image hash features from latent space image features and text hash features from latent space text features.

[0057] S7 optimizes the image autoencoder and text autoencoder parameters based on the continuous latent space image features, continuous latent space text features, image hash features, and text hash features according to the objective function proposed in this paper.

[0058] The preset feature extraction network includes: an image feature extraction network consisting of the first 7 fully connected layers of a VGG16 network, and a text feature extraction network consisting of a general sentence encoder.

[0059] The image variational autoencoder and the text variational autoencoder each include an encoder and a decoder; the encoder and decoder each include three layers, with a Leaky-Rule layer and a Batch-Norm layer inserted between the two layers.

[0060] The constructed loss functions include intra-modal reconstruction loss function, inter-modal reconstruction loss function, clustering loss function, modal alignment loss, and reconstruction consistency loss function.

[0061] The intra-modal reconstruction loss function includes

[0062] Optionally, the intra-modal reconstruction loss function The format is as follows:

[0063]

[0064] Where x I and z I These represent the original feature vector in the image modality and the continuous latent space feature vector obtained after the image autoencoder, respectively. The continuous latent space feature distribution obtained from the original feature distribution in the representative image modality. The approximate distribution of z I =μ I +σ I ⊙∈, where μ I and σ I The vectors obtained through the encoder in the image autoencoder represent the mean vector and standard deviation vector of the multivariate normal distribution, respectively. The encoder parameter is φ I , The prior distribution in the continuous latent space of the image modality is also assumed to be a normal distribution. Kullback-Leibler (KL) divergence represents the ratio between the prior distribution and the approximate distribution of the posterior distribution of latent space features in an image modality. This indicates the expected value, and log(·) represents the logarithmic operation with the natural constant as the base. This represents the original feature distribution obtained from the continuous latent space feature distribution in the image modality. The experiment uses a decoder from an image autoencoder, with the decoder's parameters being θ. I , It represents the lower bound of the maximum likelihood estimate of the original feature distribution reconstructed under the image modality;

[0065] x T and z T These represent the original feature vector in the text modality and the continuous latent space feature vector obtained after the image autoencoder, respectively. The continuous latent space feature distribution obtained from the original feature distribution in the text modality represents the continuous latent space feature distribution. The approximate distribution of z T =μ T +σ T ⊙∈, where μ T and σ T The vectors obtained through the encoder in the text autoencoder represent the mean vector and standard deviation vector of the multivariate normal distribution, respectively. The encoder parameter is φT , The prior distribution in the continuous latent space of the text modality is also assumed to be a normal distribution. Kullback-Leibler (KL) divergence represents the distance between the prior distribution and the approximate distribution of the posterior distribution of latent space features in the text modality. This represents the original feature distribution obtained from the continuous latent space feature distribution in the text modality. The experiment uses a decoder from an image autoencoder, with the decoder parameter being φ. T , It represents the lower bound of the maximum likelihood estimate of the original feature distribution reconstructed under the text modality.

[0066] Optionally, the inter-modal reconstruction loss function The format is as follows:

[0067]

[0068]

[0069] Optionally, the clustering loss function includes

[0070] Modal clustering loss function

[0071]

[0072] and The formula is as follows:

[0073]

[0074]

[0075] and The formula is as follows:

[0076]

[0077]

[0078] in, α represents the probability that the i-th sample is assigned to the j-th cluster center; α is the degree of freedom parameter of the student distribution. It is a set of centroid vectors for clustering in the image modality, which is initialized by K-means clustering and updated during training; Let represent the probability that the i-th sample belongs to the j-th cluster center, and h represent the h-th cluster center; This represents the probability that the i-th sample is assigned to the j-th cluster center in the text modality; It is a set of cluster centroid vectors in the text modality, which is initialized by K-means clustering and updated during training; This represents the probability that the i-th sample belongs to the j-th cluster center in the text modality.

[0079] Optionally, the modal alignment loss The format is as follows:

[0080]

[0081] Among them ||·|| F Let ||·||2 be the Frobenius norm of the matrix, and ||·||2 be the 2-norm of the matrix.

[0082] Optionally, the reconstruction consistency loss function The format is as follows:

[0083]

[0084] in

[0085]

[0086] ||·|| represents the magnitude of the vector, and (·) T Let be the transpose of a matrix or vector, cos(·,·) be the cosine similarity between vectors, and tanh(·) be the tangent activation function; λ and γ are preset weight parameters, n is the preset number of training samples per run, and β is a clustering density hyperparameter. β > 1 indicates that the similarity score of hash code pairs needs to be greater than 1, which correspondingly makes the nodes in the Hamming space dense, while β < 1 indicates that the similarity score of hash code pairs needs to be less than 1, which correspondingly makes the nodes in the Hamming space sparse. As the number of training steps increases, κ tends to infinity, which can make the tangent activation function gradually become the sign function. Let and The elements in the array gradually move towards +1 and -1. and It is approximately a binary vector.

[0087] Optionally, the loss function The format is as follows:

[0088]

[0089] Where ∈1, ∈2, ∈3 and ∈4 are hyperparameters of the balance loss.

[0090] In the description of this invention, it should be noted that the terms "upper," "lower," etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication between two elements. For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances.

[0091] It should also be noted that in the description of this invention, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0092] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. It is impossible to exhaustively list all the implementation methods here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. A cross-modal hash retrieval method based on a dual variational autoencoder, characterized in that, The method includes: The image modal information and text modal information of the sample are input into a preset feature extraction network to obtain the original image features and text features; Based on features in the image modality and features in the text modality, we construct image variational autoencoder models and text variational autoencoder models, embedding the original image features and original text features into a common latent space; Construct a loss function to provide constraints for the hash learning of the feature extraction network; The constructed loss functions include intra-modal reconstruction loss function, inter-modal reconstruction loss function, clustering loss function, modal alignment loss, and reconstruction consistency loss function; The intramodal reconstruction loss function for Where x I and z I These represent the original feature vector in the image modality and the continuous latent space feature vector obtained after the image autoencoder, respectively. The continuous latent space feature distribution obtained from the original feature distribution in the representative image modality. The approximate distribution of z I =μ I +σ I ⊙∈, where μ I and σ I The vectors obtained through the encoder in the image autoencoder represent the mean vector and standard deviation vector of the multivariate normal distribution, respectively. The encoder parameter is φ I , The prior distribution in the continuous latent space of the image modality is also assumed to be a normal distribution. Kullback-Leibler (KL) divergence represents the ratio between the prior distribution and the approximate distribution of the posterior distribution of latent space features in an image modality. This indicates the expected value, and log(·) represents the logarithmic operation with the natural constant as the base. This represents the original feature distribution obtained from the continuous latent space feature distribution in the image modality. The experiment uses a decoder from an image autoencoder, with the decoder's parameters being θ. I , It represents the lower bound of the maximum likelihood estimate of the original feature distribution reconstructed under the image modality; x T and z T These represent the original feature vector in the text modality and the continuous latent space feature vector obtained after the image autoencoder, respectively. The continuous latent space feature distribution obtained from the original feature distribution in the text modality represents the continuous latent space feature distribution. The approximate distribution of z T =μ T +σ T ⊙∈, where μ T and σ T The vectors obtained through the encoder in the text autoencoder represent the mean vector and standard deviation vector of the multivariate normal distribution, respectively. The encoder parameter is φ T , The prior distribution in the continuous latent space of the text modality is also assumed to be a normal distribution. Kullback-Leibler (KL) divergence represents the distance between the prior distribution and the approximate distribution of the posterior distribution of latent space features in the text modality. This represents the original feature distribution obtained from the continuous latent space feature distribution in the text modality. The experiment uses a decoder from an image autoencoder, with the decoder parameter being φ. T , It represents the lower bound of the maximum likelihood estimate of the original feature distribution reconstructed under the text modality; The intermodal reconstruction loss function The format is as follows:

2. The method according to claim 1, characterized in that, The preset feature extraction network includes: an image feature extraction network consisting of the first 7 fully connected layers of a VGG16 network, and a text feature extraction network consisting of a general sentence encoder.

3. The method according to claim 1, characterized in that, The image variational autoencoder and the text variational autoencoder each include an encoder and a decoder; the encoder and decoder each include three layers, with a Leaky-Rule layer and a Batch-Norm layer inserted between the two layers.

4. The method according to claim 1, characterized in that, The clustering loss function includes Modal clustering loss function in, This represents the probability that the i-th sample is assigned to the j-th cluster center in the image modality. This represents the probability that the i-th sample belongs to the j-th cluster center in the image modality; This represents the probability that the i-th sample is assigned to the j-th cluster center in the text modality; This represents the probability that the i-th sample belongs to the j-th cluster center in the text modality.

5. The method according to claim 1, characterized in that, The modal alignment loss The format is as follows: Among them ||·|| F Let ||·||2 be the Frobenius norm of the matrix, and ||·||2 be the 2-norm of the matrix. and Let and represent the mean vector and standard deviation vector of the multivariate normal distribution generated by the encoder under image modality, respectively. and These represent the mean vector and standard deviation vector of the multivariate normal distribution generated by the encoder in the text modality for the i-th sample, respectively.

6. The method according to claim 1, characterized in that, The reconstruction consistency loss function The format is as follows: in ||·|| represents the magnitude of the vector, and (·) T Let be the transpose of a matrix or vector, cos(·,·) be the cosine similarity between vectors, and tanh(·) be the tangent activation function; λ and γ are preset weight parameters, n is the preset number of training samples per run, and β is a clustering density hyperparameter. β > 1 indicates that the similarity score of hash code pairs needs to be greater than 1, which correspondingly makes the nodes in the Hamming space dense, while β < 1 indicates that the similarity score of hash code pairs needs to be less than 1, which correspondingly makes the nodes in the Hamming space sparse. As the number of training steps increases, κ tends to infinity, which can make the tangent activation function gradually become the sign function. Let and The elements in the array gradually move towards +1 and -1. and It is approximately a binary vector.

7. The method according to claim 6, characterized in that, loss function The format is as follows: Where ∈1, ∈2, ∈3 and ∈4 are hyperparameters of the balance loss.

Images