Depth semantic guided unsupervised multi-modal Hash retrieval method

Abstract

The invention belongs to the field of multi-modal information retrieval, and discloses a deep semantic guided unsupervised multi-modal Hash retrieval method, which is characterized in that a teacher model adopts powerful modal interaction information provided by a VLP model, a similar matrix and a graph convolutional neural network are constructed, and the semantic gap between a real value space and a Hamming space is made up; rich semantic information in the teacher model is transmitted to the student model in a knowledge distillation mode, and the student model is optimized, so that efficient learning of the small model is promoted; the student model inherits the advantages of the teacher model and avoids hyper-parameter adjustment pressure of a large-scale model, so that the method is more suitable for a large-scale data scene in practical application. A Hamming distance optimization strategy is designed, it is ensured that the complex semantic relation in the real value space is reasonably quantified and expressed in the Hamming space, and excessive compression from the real value space to the Hamming space is avoided.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal information retrieval, and specifically relates to an unsupervised multimodal hash retrieval method guided by deep semantics. Background Art

[0002] With the explosion of information and the rise of cloud computing, cross-modal hashing methods have been favored by more and more researchers in the process of efficiently retrieving large amounts of heterogeneous and high-dimensional data on the Internet due to their low storage cost and fast retrieval. Hashing technology can project high-dimensional data into compact binary hash codes. With it, the most time-consuming semantic similarity calculation in the multimedia retrieval process can be significantly accelerated by fast Hamming distance calculation, while the storage cost can be greatly reduced through binary embedding. It is also important to note that almost all existing cross-modal hashing techniques uniformly project the learned feature vectors into the Hamming space for semantic and modality alignment to minimize the modality gap.

[0003] According to the features adopted in the training phase, current cross-modal hashing techniques can be divided into non-deep and deep techniques. The features learned by non-deep methods are manually extracted, and the semantic information of the original data is often not comprehensive enough to generate the best binary hash code, which limits the retrieval performance. Compared with non-deep hashing methods, deep hashing methods use neural networks to fully and accurately capture the semantic content of the original data, improving the discriminative ability of recorded features. In view of the powerful nonlinear representation ability of deep neural networks, many studies on deep-based cross-modal hashing techniques have been published in recent years. For example, the paper "Song G, Zhang W, Wang B. Deep cross-modal hashing with contrast learning and feature fusion" proposes a method that combines mid-level features of image and text data, which alleviates the semantic loss in the feature learning process and pays attention to the semantic correlation between and within modes. "Peng Y, Jiang X, Zhang N, et al. Graph Convolutional Networks Based Muti-Label Deep Cross-Modal Hashing" introduces a graph convolutional network to capture the class correlation of multiple labels and supervises the training of hash graphs to avoid the loss of rich semantic information in labels. However, there are still some problems and challenges in the current deep cross-modal hashing technology: many deep cross-modal hashing technologies prioritize feature extraction in a single modality, ignoring the correlation between modalities, resulting in insufficient feature representation. The emergence of the VLP model (Radford A, Kim JW, Hallacy C, et al. Learning transferable visual models from natural language supervision) provides a new path to solve these problems. The VLP (visual-language pre-training) model can capture deeper semantic associations between images and texts by jointly training on large-scale multimodal datasets. This model not only considers feature extraction within each modality, but also effectively fuses data from different modalities through a shared representation space, thereby achieving high consistency and semantic alignment of cross-modal features. Although the VLP model used has excellent multimodal processing capabilities, its large number of requirements for hyperparameter adjustment and complex model structure or high computational cost make it unusable for large-scale unsupervised retrieval tasks. In addition, the spatial gap between the real-valued space and the Hamming space will have a profound impact on the efficiency of semantic conversion. In many practical applications, such as image retrieval, text retrieval, etc., data usually exists in the form of real-valued vectors in a continuous real-valued space.The real-valued vector space is continuous and has a high dimension, while the Hamming space is a discrete space composed of binary codes, which is usually used for compact representation in storage and calculation. When data is mapped from the real-valued space to the Hamming space, due to the difference in their properties, semantic information may be lost, affecting the efficiency of the model in retrieval or classification tasks. Therefore, how to preserve the semantic information of the original data as much as possible during the conversion process is an important research issue. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a deep semantic-guided unsupervised multimodal hash retrieval method, constructs a knowledge distillation technology in a deep cross-modal hash model (abbreviated as: SKDH model), which mainly includes a teacher model and a student model. The teacher model uses the powerful modal interaction information provided by the VLP model to construct a similarity matrix and a graph convolutional neural network (GCN network), which makes up for the semantic gap between the real-valued space and the Hamming space; the rich semantic information in the teacher model is transferred to the student model by the knowledge distillation method, and the student model is optimized by extracting hash codes and fine-grained semantic information from the large model, thereby promoting the efficient learning of the small model, and then the student model can inherit the excellent characteristics of the teacher model without the need for a complex model structure or high computational cost; while inheriting the advantages of the teacher model, the student model avoids the pressure of hyperparameter adjustment of the large-scale model, so that it is more suitable for large-scale data scenarios in practical applications. An optimization strategy is designed for the Hamming distance to ensure that the complex semantic relationship in the real-valued space is reasonably quantified and expressed in the Hamming space, avoiding excessive compression from the real-valued space to the Hamming space.

[0005] The deep semantics-guided unsupervised multimodal hash retrieval method of the present invention comprises the following steps:

[0006] Step 1: Obtain a multimodal dataset and perform standardization on the multimodal image dataset;

[0007] Step 2: construct a SKDH model, wherein the teacher model in the SKDH model extracts multimodal features of image data and text data in the multimodal dataset through the VLP model to capture the semantic associations between different modalities;

[0008] Step 3: Construct a similarity matrix based on the extracted multimodal features and input it into the graph convolutional neural network GCN to generate a binary hash code;

[0009] Step 4: Enhance the capabilities of the student model in the SKDH model through knowledge distillation, construct a loss function, and achieve accurate semantic alignment between different modalities;

[0010] Step 5: Based on the student model, optimize the Hamming distance allocation strategy for generating hash codes, construct a comprehensive objective function, and achieve accurate retrieval.

[0011] Furthermore, step 1 is specifically as follows:

[0012] Step 1-1: Obtain a multimodal dataset, which includes paired image and text data Where v represents the image mode, t represents the text mode, and v i represents the i-th image data, t i represents the i-th text data, n is the total number of image-text pairs, represents the real number space, d v Represents the dimension of image modality data; d t Represents the dimension of text modality data;

[0013] Step 1-2: Perform data preprocessing on the multimodal dataset, scale the image to a size of 256x256, keep the aspect ratio of the image unchanged to help standardize the size of the input image, and then randomly crop a 224×224 pixel area from the resized image, and normalize the image using the given mean and standard deviation.

[0014] Furthermore, step 2 is specifically as follows: using the VLP visual model to extract features of n image data in the multimodal dataset to form image features At the same time, the VLP text model is used to extract the features of n text data in the multimodal dataset to form text features Through the VLP visual model and VLP text model, the teacher model captures the deep semantic associations between images and texts, providing feature representations rich in semantic information for subsequent hash code learning.

[0015] Furthermore, step 3 is specifically as follows:

[0016] Step 3-1: Based on image features and text features Constructing the similarity matrix consists of two parts. The first part captures the intra-modal similarity information including the image self-similarity matrix S v and the text self-similarity matrix S t , revealing data relationships within the visual and textual modalities, respectively, The second part contains cross-modal similarity information, which is used to bridge the semantic gap between the visual modality and the textual modality, and contains the cross-modal similarity matrix S inter , in Representative calculation and The cosine similarity between two features;

[0017] S v , St , S inter The three similarity matrices are combined to construct the learning guidance matrix S G , the formation process is as follows:

[0018] S G =λ1S v +λ2S t +λ3S inter ∈[-1,+1] m×m

[0019] stλ1,λ2,λ3≥0,λ1+λ2+λ3=1

[0020] Where λ1, λ2, and λ3 are adjustable hyperparameters that adjust the relative weights assigned to similarity information derived from different modalities, and m is the learning guidance matrix S G Dimensions;

[0021] Step 3-2: Image features and text features Input graph convolutional neural network GCN to get feature Z * , and through the similarity matrix S G Guide the learning of graph convolutional neural networks; specifically:

[0022] The graph convolutional neural network uses the similarity matrix generated in the previous step to capture the semantic associations between data samples, and integrates this semantic information into the learning process of the hash code through multi-layer graph convolution operations, thereby converting image features into and text features Convert to binary hash code

[0023]

[0024] Where α is a hyperparameter that controls the number of iterations, tanh is a nonlinear activation function, and c is a binary hash code The dimension, Z * is the image feature and text features It is obtained through GCN.

[0025] Furthermore, step 4 is specifically as follows:

[0026] Step 4-1: The visual model of the student model uses the VGG-16 visual model, and the text model uses the fully connected layer text model, namely the MLP text model, from the multimodal dataset Extract features from

[0027] The VGG-16 visual model takes the input image data v iConvert to feature vector B v , while the MLP text model converts the input text data t i Converted into text vector B t ; In order to improve the quality of the student model hash code, the knowledge distillation method is used to form a binary representation alignment loss model called As shown below:

[0028]

[0029] in represents the binary representation alignment loss, represents the F-norm of the matrix, and denote the image and text hash codes of the teacher model respectively;

[0030] Step 4-2: Calculate cross-modal distillation losses separately and intra-modal distillation loss To guide the optimization of the hash code generator in the student model, as follows:

[0031]

[0032] In the formula, represents the Hadamard product operation, Represents the cross-modal similarity graph of the student model, which is composed of the image hash code B of the student model v and the transpose of the text hash code of the student model Point multiplication; Represents the teacher model image self-similarity graph, which is composed of the image hash code of the teacher model and the transpose of the image hash code of the teacher model Point multiplication; They represent the text self-similarity graph of the teacher model, respectively, and are composed of the text hash code of the teacher model and the transpose of the text hash code of the teacher model Point multiplication; μ is a hyperparameter used to flexibly adjust the quantization range of a given matrix.

[0033] Furthermore, step 5 is specifically as follows:

[0034] Step 5-1: Define the form of Hamming distance as λ ij =c / 2(1-S ij ), where c represents the hash code length, S ij Represents the similarity matrix S G The pairwise similarity between the i-th and j-th instances in , λ ij represents the newly established Hamming distance;

[0035] The semantic channel width is introduced to redistribute the Hamming distance for similar and dissimilar data pairs as follows:

[0036]

[0037] Where ⊙ represents the dot product operation of the matrix, σ represents the maximum value function, Represent the lower and upper bounds of the semantic channel respectively, and are Construct upper bound matrices separately and the lower bound matrix And provide corresponding Hamming constraints for data pairs; W l is the lower bound weight matrix, W u is the upper bound weight matrix;

[0038] When S ij = 0, assuming that any two dissimilar samples have corresponding hash codes b i and In this case, the lower bound weight matrix W l Set to 0, the upper bound weight matrix W u Set to α; to ensure the dissimilarity of hash codes without being too strict with space constraints, it is expressed as d H represents the Hamming distance between parameters;

[0039] When S ij ∈(0,1), assuming that any two samples are not completely similar, the corresponding hash codes are b i and Lower bound weight matrix W l and the upper bound weight matrix W u are all set to 1; the distance of the hash code is constrained by the upper and lower bounds at the same time, expressed as

[0040] When S ij = 1, assuming that any two completely similar samples have corresponding hash codes b i and Lower bound weight W l Set to β, the upper bound weight W u Set to 0; only pay attention to the upper bound to ensure that the Hamming distance between completely semantic positive samples is minimized, expressed as

[0041] Step 5-2: Construct a comprehensive objective function As shown below:

[0042]

[0043] sB v ,Bt ∈{-1,+1} m×c

[0044] Where μ1, μ2, μ3 and μ4 represent the parameters for balancing the loss terms.

[0045] The beneficial effects of the present invention are:

[0046] (1) Although the traditional VLP model has the ability to extract visual and language modal features, due to the characteristics of the pre-trained model, it may have certain deficiencies in cross-modal feature fusion, especially in the modeling of deep semantic relationships between multimodal data. The method described in the present invention uses the powerful modal interaction information provided by the VLP model to establish a teacher model. The pre-training of the VLP (visual-language pre-training) model on a large-scale data set can fully capture the complex semantic associations between images and texts. The teacher model uses its complex network structure and stronger representation ability to deeply mine the data in the real-valued space and construct a comprehensive similarity matrix, which effectively reflects the intrinsic associations between different modal data, not only retains the original semantic information of the data, but also effectively bridges the gap between the real-valued space and the Hamming space; through these features and similarity information, further combined with the graph convolutional neural network (GCN) framework, by aggregating the features of similar data, the robustness and semantic expression ability of the generated hash code are improved, thereby generating a more robust and semantically rich binary hash code, which makes up for the limitations of the VLP model in processing complex multimodal interactions. The present invention not only improves the effect of feature fusion, but also significantly enhances the stability and semantic fidelity of the hash code generated by the model, effectively making up for the insufficient performance of the VLP model in cross-modal retrieval, so that the overall model has stronger performance and higher retrieval accuracy in complex multimodal data processing.

[0047] (2) Traditional hashing methods directly map between real-valued space and Hamming space, which easily leads to the loss of semantic information of data during the conversion process, resulting in the inability to fully align features between different modalities, thus affecting the final retrieval effect. In order to reduce the semantic loss caused by the spatial gap generated by the simple conversion from real-valued space to Hamming space in the student model, the method of the present invention proposes to introduce a refined Hamming distance allocation strategy, which optimizes the allocation of Hamming distance and flexibly adjusts the Hamming distance according to the semantic similarity between data pairs, so that similar data pairs have a shorter distance in the Hamming space, while dissimilar data pairs have a longer distance. This refined Hamming distance allocation strategy can not only effectively utilize the capacity of the Hamming space and reduce the loss of semantic information during the discretization process, but also maximize the retention of semantic relationships between data by reducing the distance deviation of similar instances in the Hamming space, thereby significantly improving the accuracy of cross-modal retrieval. By combining the similarity matrix and the global semantic information of the data captured by the graph convolutional neural network (GCN), the semantic relationship between the data can be accurately reflected, ensuring that the complex semantic relationship in the real-valued space is reasonably quantified and expressed in the Hamming space, avoiding excessive compression from the real-valued space to the Hamming space, reducing the semantic loss caused by simple space conversion, and making the generated hash code more in line with actual semantic needs.

[0048] (3) The knowledge distillation technology adopted by the present invention effectively transfers the rich semantic information in the large-scale pre-trained teacher model to the lighter student model, so that the student model can inherit the powerful feature extraction ability of the teacher model while maintaining computational efficiency. Even under unsupervised conditions, the student model can still learn high-quality feature representations close to the teacher model, thereby generating accurate and semantically rich binary hash codes; it not only improves the performance of the student model when processing multimodal data, but also significantly reduces the computing resources and time cost required by the model, making it possible for the system to efficiently retrieve large-scale data sets. The application of this technology ensures that the system can maintain efficient retrieval performance under limited resources, while greatly reducing the hardware requirements and energy consumption in practical applications, expanding the application scope and operability of the system, especially in large-scale unsupervised scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Flowchart of the method of the present invention. DETAILED DESCRIPTION

[0050] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0051] like Figure 1As shown, the deep semantics-guided unsupervised multimodal hash retrieval method of the present invention comprises the following steps:

[0052] Step 1: obtaining a multimodal dataset and performing standardization processing on the multimodal image dataset;

[0053] Step 2: In the SKDH model, the teacher model uses the VLP model to extract multimodal features; the VLP visual model and text model will extract high-level semantic features from the visual data and text data respectively. These features can capture the complex semantic associations between different modalities and provide semantically enhanced information support for the hash code learning process, so as to ensure that the semantic consistency of the data can be better maintained and expressed during the hash code generation process.

[0054] Step 3: Construct a similarity matrix and graph convolution auxiliary framework by extracting multimodal features from the VLP model to provide high-quality targets for hash code learning. The similarity matrix is ​​responsible for guiding the graph convolution network and the student model, and the graph convolution auxiliary framework enhances the representation ability of the hash code by aggregating the features of similar data;

[0055] Step 4. Establish a student model and enhance the capabilities of the student model through knowledge distillation. In the hash code learning process, in order to improve the performance of the student model, knowledge distillation technology will be used to transfer rich semantic information from the teacher model to the student model. The teacher model uses its powerful feature extraction and expression capabilities to generate high-quality hash codes and semantic guidance matrices. On this basis, the student model continuously optimizes its own hash function by imitating the hash codes and similarity matrices output by the teacher model. In this process, by minimizing the difference between the hash code generated by the student model and the hash code of the teacher model, as well as the cross-modal and intra-modal similarity losses, the student model can gradually learn more detailed semantic features and similarity relationships, thereby achieving retrieval performance close to that of the teacher model under a smaller network structure. In the end, knowledge distillation effectively improves the discrimination ability and learning efficiency of the student model, so that it can still maintain efficient and accurate retrieval performance when processing large-scale multimodal data;

[0056] Step 5: In order to solve the spatial gap and semantic loss problems caused by the conversion from real-valued space to Hamming space in the student model, the system maximizes the use of the Hamming space by introducing semantic channels, and solves the compression problem of the solution space by optimizing the distribution of Hamming distance. On this basis, the system combines the aforementioned multiple loss functions to form the final comprehensive objective function.

[0057] Furthermore, step 1 is specifically as follows:

[0058] Step 1-1: Obtain a multimodal dataset, which includes paired image and text data Where v represents the image mode, t represents the text mode, and v i represents the i-th image data, t i represents the i-th text data, n is the total number of image-text pairs, represents the real number space, d v Represents the dimension of image modality data; d t Represents the dimension of text modality data.

[0059] Step 1-2: Preprocess the multimodal dataset, resize the shorter side of the image to 256 pixels, keep the aspect ratio unchanged to help standardize the size of the input image, and randomly crop a 224×224 pixel area from the resized image to increase the diversity of the training data while unifying the input size. Use the given mean and standard deviation to normalize the image. Normalization is to make the model training more stable, faster, and more robust.

[0060] Furthermore, step 2 is specifically as follows:

[0061] The present invention uses the VLP visual model in the VLP model to extract the features of n image data in the multimodal data set to form an image feature At the same time, the VLP text model in the VLP model is used to construct text features from the features of n text data in the multimodal dataset Through these VLP models, the teacher model can capture the deep semantic association between images and texts, providing feature representations rich in semantic information for subsequent hash code learning. and text features It will be further used to construct a similarity matrix and a graph convolution auxiliary framework to obtain a high-quality semantic guidance matrix and optimize the hash code generation process.

[0062] Furthermore, step 3 is specifically as follows:

[0063] Step 3-1: Different from the traditional unsupervised method that directly uses the extracted multimodal features to generate the final hash representation, the present invention makes full use of the rich semantic information embedded in these features to construct a reliable learning guidance similarity matrix. The similarity matrix consists of two main parts: the first part captures the intra-modal similarity information and contains the image self-similarity matrix S v and the text self-similarity matrix S t , revealing data relationships within the visual and textual modalities, respectively, The second part contains cross-modal similarity information, which is used to bridge the semantic gap between the visual modality and the textual modality, including the cross-modal similarity matrix S inter , in Representative calculation and The cosine similarity between two features is usually used to measure the similarity between two vector directions.

[0064] Furthermore, S v , S t , S inter The three similarity matrices are combined to construct a robust learning guidance matrix S G , the formation process is as follows:

[0065] S G =λ1S v +λ2S t +λ3S inter ∈[-1,+1] m×m

[0066] stλ1,λ2,λ3≥0,λ1+λ2+λ3=1

[0067] Wherein, λ1, λ2, and λ3 are all adjustable hyperparameters, and the present invention proposes to set the values ​​to 0.5, 0.3, and 0.2 respectively, which adjust the relative weights assigned to the similarity information derived from different modalities, and m is the learning guidance matrix S G In this way, the present invention can construct a semantically rich similarity matrix, effectively encapsulate the correlation information between instances, and help to efficiently learn hash representation.

[0068] Step 3-2: Further, the image features and text features Input the graph convolutional neural network (GCN) auxiliary framework to obtain feature Z * And through the similarity matrix S G Guide the learning of the graph convolutional neural network auxiliary framework. This framework enhances the representation ability of hash codes by aggregating the features of similar data. Specifically, the graph convolutional neural network uses the similarity matrix generated in the previous step to capture the semantic associations between data samples, and integrates this semantic information into the learning process of hash codes through multi-layer graph convolution operations, thereby integrating image features. and text features Transformed into a more robust and semantically rich binary hash code

[0069]

[0070] Where α is a hyperparameter that controls the number of iterations, tanh is a nonlinear activation function, and c is a binary hash code The dimension, Z * is the image feature and text features It is obtained through the GCN-assisted framework. In order to reduce the information loss inherent in binarization, we adopt an iterative optimization strategy. Specifically, when approaching infinity, the function becomes steeper and steeper, forcing the output values ​​to tend to the extremes of the range, i.e. -1 and +1. The iterative quantization process gives the model increasing robustness, enabling it to effectively filter out irrelevant information and extract the most significant features from noisy and changing input data.

[0071] During the teacher model training, the present invention builds a comprehensive similarity matrix S G , and a robust teacher model hash code is learned through an auxiliary network and Furthermore, step 4 is specifically as follows:

[0072] Step 4-1: In the simple student model, the student model uses a lightweight model to extract the multimodal dataset Specifically, the student model’s visual model uses the VGG-16 visual model. The text model uses the fully connected layer text model (MLP text model for short) to process text data and extract corresponding text features. Both the VGG-16 visual model and the MLP text model can be directly called from the pytorch library. The VGG-16 visual model converts the input image data v i Convert to feature vector B v , while the MLP text model converts the input text data t i Converted into text vector B t To improve the quality of the student model hash code, we adopt a multi-faceted approach to transfer rich semantic information from the teacher model to the student model: We use knowledge distillation to construct a binary representation alignment loss model called As shown below.

[0073]

[0074] in represents the binary representation alignment loss, represents the F-norm of the matrix, and denote the image and text hash codes of the teacher model, respectively, Useful for delivering fine-grained hash features.

[0075] Step 4-2: In addition, the present invention also calculates the cross-modal distillation loss and intra-modal distillation loss To guide the optimization of the hash code generator in the student model, as shown below.

[0076]

[0077] In the formula, represents the Hadamard product operation, Represents the cross-modal similarity graph of the student model, which is composed of the image hash code B of the student model v and the transpose of the text hash code of the student model Dot multiplication. Represents the teacher model image self-similarity graph, which is composed of the image hash code of the teacher model and the transpose of the image hash code of the teacher model Dot multiplication. They represent the text self-similarity graph of the teacher model, respectively, and are composed of the text hash code of the teacher model and the transpose of the text hash code of the teacher model Dot multiplication. μ is a hyperparameter used to flexibly adjust the quantization range of a given matrix.

[0078] Through the above process, the student model can gradually inherit rich semantic information from the teacher model, so that the generated hash code can maintain high-quality representation capabilities in both cross-modal and intra-modal tasks. As the knowledge distillation process proceeds, the hash code generator of the student model is continuously optimized, gradually narrowing the gap with the output of the teacher model, thereby achieving more accurate semantic alignment between different modalities.

[0079] In step 5, the student model directly maps the extracted real-valued features to the Hamming space to generate a hash representation. The spatial gap inherent in simple space transformation may affect the accuracy of the semantic relationship description to a certain extent. Traditional strategies, such as minimizing quantization error, cannot compensate for the semantic loss in the transformation between the real-valued space and the Hamming space, and may even lead to decompression problems. In order to address these challenges, the present invention redefines the Hamming distance in the form of λ ij =c / 2(1-S ij ), where c represents the hash code length, S ij Represents the similarity matrix S G The pairwise similarity between the i-th and j-th instances in , λ ij represents the newly established appropriate Hamming distance.

[0080] In addition, the present invention introduces semantic channel width to maximize the utilization of the Hamming space. Specifically, the present invention reallocates more suitable Hamming distances for similar and dissimilar data pairs, as shown below:

[0081]

[0082] Where ⊙ represents the dot product operation of the matrix, σ represents the maximum value function, Represent the lower and upper bounds of the semantic channel respectively, and are Construct upper bound matrices separately and the lower bound matrix And provide corresponding Hamming constraints for data pairs, W l is the lower bound weight matrix, W u is the upper bound weight matrix. ij = 0, the present invention is to better illustrate S ij , assuming that any two dissimilar samples have hash codes b i and In this case, the lower bound weight matrix W l Set to 0, the upper bound weight matrix W u The present invention aims to ensure the dissimilarity of hash codes without being subject to overly strict space restrictions, which can be expressed as d H represents the Hamming distance between parameters. ij ∈(0,1), the present invention is to better explain S ij , assuming that any two samples are not completely similar, the hash codes corresponding to these two samples are b i and Lower bound weight matrix W l and the upper bound weight matrix W u are all set to 1. The present invention uses upper and lower bounds to constrain the distance of hash codes at the same time, expressed as When S ij =1, the present invention is to better illustrate S ij , assuming that any two completely similar samples, the hash codes corresponding to these two completely similar samples are b i and Lower bound weight W l Set to β, the upper bound weight W u Set to 0. The present invention can only focus on the upper bound to ensure that the Hamming distance between the complete semantic positive samples is minimized, which is expressed as

[0083] By utilizing the comprehensive similarity matrix S G ,The present paper optimizes the Hamming distance distribution for generating hash codes, thereby maximizing the utilization of the Hamming space and alleviating the ,compression problem of the solution space. In addition, by utilizing the refined Hamming distance to establish ,semantic channels, the oscillation problem inherent in the loss function is ,effectively alleviated.

[0084] In the above optimization process, the present invention effectively improves the quality of hash code generation and maximizes the utilization of Hamming space by introducing semantic channels and refined Hamming distance allocation. In order to further consolidate this process, the present invention combines the aforementioned multiple loss functions and finally forms the comprehensive objective function of the proposed framework of the present invention As shown below:

[0085]

[0086] sB v ,B t ∈{-1,+1} m×c

[0087] Among them, μ1, μ2, μ3 and μ4 represent the parameters for balancing the loss terms, and the present invention proposes to set the values ​​to 0.1, 0.1, 1, and 1 respectively. By minimizing the above comprehensive loss function, the student model can effectively distill semantic knowledge from the teacher model and optimize its own learning parameters at the same time. This optimization process not only ensures the efficient representation of hash codes in the Hamming space, but also improves the overall performance of cross-modal retrieval tasks.

[0088] The search process of the present invention is:

[0089] First, a set of multimodal data (such as a pair of images and text) is input into the system. The data of each modality is input into the corresponding teacher model, including the VLP visual model (for processing images) and the VLP text model (for processing text), to extract visual features and text features respectively. These features are high-dimensional and contain rich semantic information.

[0090] Next, the extracted multimodal features are used to construct a similarity matrix, which includes intra-modal similarity and cross-modal similarity. These similarity matrices help capture the relationship between images, texts, and images. The similarity matrix is ​​input into the graph convolutional neural network (GCN), which enhances the representation of hash codes by aggregating features of similar data, thereby generating more robust and semantically rich binary hash codes.

[0091] In the process of generating hash codes, in order to solve the problem of semantic information loss that may be caused by the conversion from real-valued space to Hamming space in traditional methods, the system introduces an optimized Hamming distance allocation strategy. This strategy redefines the Hamming distance to assign more appropriate Hamming distances to similar and dissimilar data pairs. Specifically, the system assigns shorter Hamming distances to similar data pairs and longer Hamming distances to dissimilar data pairs based on the semantic information of the similarity matrix. This refined Hamming distance adjustment not only ensures the representation ability of the hash code in the Hamming space, but also maximizes the utilization of the Hamming space, thereby avoiding the problem of knowledge space compression.

[0092] In actual retrieval tasks, query data (such as query images or query text) will also be converted into hash codes, and the Hamming distance will be calculated with the hash codes stored in the database. Since the system optimizes the Hamming distance, the distance calculation between the query hash code and the database hash code can more accurately reflect the semantic relationship between the data. By calculating the optimized Hamming distance, the system can quickly find the closest match to the query data and provide more accurate retrieval results.

[0093] Simulation conditions

[0094] In this system method experiment, all experiments were conducted on a workstation with Win11 system. The simulation software used was Pycharm, the version of Pytorch was 2.4.0, the version of CUDA was 12.4, and the version of Python was 3.11.8. All functions and libraries used were public and callable. The experiments were conducted on the MIRFlickr, NUS-WIDE, and MSCOCO datasets.

[0095] Simulation Results

[0096] The MAP performance of the SKDH of the present invention on the MIRFlickr, NUS-WIDE and MSCOCO datasets is shown in Table

[0097] Table 1 MAP performance on the MIRFlickr dataset

[0098]

[0099] Table 2 MAP performance on NUS-WIDE dataset

[0100]

[0101]

[0102] Table 3 MAP performance on the MSCOCO dataset

[0103]

[0104] Comprehensive experiments on three benchmark datasets consistently demonstrate the excellent cross-modal retrieval capability of our approach, highlighting its practical applicability in handling large-scale multimodal data scenarios.

[0105] The above description is only a preferred embodiment of the present invention and is not intended to be a further limitation of the present invention. All equivalent changes made using the contents of the present specification and drawings are within the protection scope of the present invention.

Claims

1. Deep semantics-guided unsupervised multimodal hash retrieval method, characterized by: The following steps are involved: Step 1: Obtain a multimodal dataset and perform standardization on the multimodal image dataset; Step 2: construct a SKDH model, wherein the teacher model in the SKDH model extracts multimodal features of image data and text data in the multimodal dataset through the VLP model to capture the semantic associations between different modalities; Step 3: Construct a similarity matrix based on the extracted multimodal features and input it into the graph convolutional neural network GCN to generate a binary hash code; Step 4: Enhance the capabilities of the student model in the SKDH model through knowledge distillation, construct a loss function, and achieve accurate semantic alignment between different modalities; Step 5: Based on the student model, optimize the Hamming distance allocation strategy for generating hash codes, construct a comprehensive objective function, and achieve accurate retrieval.

2. The deep semantics-guided unsupervised multimodal hash retrieval method according to claim 1, characterized in that: Step 1 is as follows: Step 1-1: Obtain a multimodal dataset, which includes paired image and text data Where v represents the image mode, t represents the text mode, and v i represents the i-th image data, t i represents the i-th text data, n is the total number of image-text pairs, represents the real number space, d v Represents the dimension of image modality data; d t Represents the dimension of text modality data; Step 1-2: Perform data preprocessing on the multimodal dataset, scale the image to a size of 256x256, randomly crop a 224×224 pixel area from the resized image, and normalize the image using the given mean and standard deviation.

3. The deep semantics-guided unsupervised multimodal hash retrieval method according to claim 1, characterized in that: Step 2 is as follows: Use the VLP visual model to extract the features of n image data in the multimodal dataset to form image features At the same time, the VLP text model is used to extract the features of n text data in the multimodal dataset to form text features Through the VLP visual model and VLP text model, the teacher model captures the deep semantic associations between images and texts, providing feature representations rich in semantic information for subsequent hash code learning.

4. The deep semantics-guided unsupervised multimodal hash retrieval method according to claim 3, characterized in that: Step 3 is as follows: Step 3-1: Based on image features and text features Constructing the similarity matrix consists of two parts. The first part captures the intra-modal similarity information including the image self-similarity matrix S v and the text self-similarity matrix S t , revealing data relationships within the visual and textual modalities, respectively, The second part contains cross-modal similarity information, which is used to bridge the semantic gap between the visual modality and the textual modality, and contains the cross-modal similarity matrix S inter , in Representative calculation and The cosine similarity between two features; S v , S t , S inter The three similarity matrices are combined to construct the learning guidance matrix S G , the formation process is as follows: S G =λ1S v +λ2S t +λ3S inter ∈[-1,+1] m×m stλ1,λ2,λ3≥0,λ1+λ2+λ3=1 Where λ1, λ2, and λ3 are adjustable hyperparameters that adjust the relative weights assigned to similarity information derived from different modalities, and m is the learning guidance matrix S G Dimensions; Step 3-2: Image features and text features Input graph convolutional neural network GCN to get feature Z * , and through the similarity matrix S G Guide the learning of graph convolutional neural networks; specifically: The graph convolutional neural network uses the similarity matrix generated in the previous step to capture the semantic associations between data samples, and integrates this semantic information into the learning process of the hash code through multi-layer graph convolution operations, thereby converting image features into and text features Convert to binary hash code Where α is a hyperparameter that controls the number of iterations, tanh is a nonlinear activation function, and c is a binary hash code The dimension, Z * is the image feature and text features It is obtained through GCN.

5. The deep semantics-guided unsupervised multimodal hash retrieval method according to claim 4, characterized in that: Step 4 is as follows: Step 4-1: The visual model of the student model uses the VGG-16 visual model, and the text model uses the fully connected layer text model, namely the MLP text model, from the multimodal dataset Extract features from The VGG-16 visual model takes the input image data v i Convert to feature vector B v , while the MLP text model converts the input text data t i Converted into text vector B t ; The binary representation alignment loss model is constructed by knowledge distillation and is expressed as As shown below: in represents the binary representation alignment loss, represents the F-norm of the matrix, and denote the image and text hash codes of the teacher model respectively; Step 4-2: Calculate cross-modal distillation losses separately and the intra-modal distillation loss To guide the optimization of the hash code generator in the student model, as follows: In the formula, represents the Hadamard product operation, Represents the cross-modal similarity graph of the student model, which is composed of the image hash code B of the student model v and the transpose of the text hash code of the student model Point multiplication; Represents the teacher model image self-similarity graph, which is composed of the image hash code of the teacher model and the transpose of the image hash code of the teacher model Point multiplication; They represent the text self-similarity graph of the teacher model, respectively, and are composed of the text hash code of the teacher model and the transpose of the text hash code of the teacher model Point multiplication; μ is a hyperparameter used to flexibly adjust the quantization range of a given matrix.

6. The deep semantics-guided unsupervised multimodal hash retrieval method according to claim 5, characterized in that: Step 5 is as follows: Step 5-1: Define the form of Hamming distance as λ ij =c / 2(1-S ij ), where c represents the hash code length, S ij Represents the similarity matrix S G The pairwise similarity between the i-th and j-th instances in , λ ij represents the newly established Hamming distance; The semantic channel width is introduced to redistribute the Hamming distance for similar and dissimilar data pairs as follows: Where ⊙ represents the dot product operation of the matrix, σ represents the maximum value function, Represent the lower and upper bounds of the semantic channel respectively, and are Construct upper bound matrices separately and the lower bound matrix And provide corresponding Hamming constraints for data pairs; W l is the lower bound weight matrix, W u is the upper bound weight matrix; When S ij = 0, assuming that any two dissimilar samples have corresponding hash codes b i and Lower bound weight matrix W l Set to 0, the upper bound weight matrix W u Set to α; to ensure the dissimilarity of hash codes without being too strict with space constraints, it is expressed as d H represents the Hamming distance between parameters; When S ij ∈(0,1), assuming that any two samples are not completely similar, the corresponding hash codes are b i and Lower bound weight matrix W l and the upper bound weight matrix W u are all set to 1; the distance of the hash code is constrained by the upper and lower bounds at the same time, expressed as When S ij = 1, assuming that any two completely similar samples have corresponding hash codes b i and Lower bound weight W l Set to β, the upper bound weight W u Set to 0; We only focus on the upper bound to ensure that the Hamming distance between fully semantically positive samples is minimized, expressed as Step 5-2: Construct a comprehensive objective function As shown below: s.t.B v ,B t ∈{-1,+1} m×c Where μ1, μ2, μ3 and μ4 represent the parameters for balancing the loss terms.

Images