Comparative learning unsupervised cross-modal hash retrieval algorithm based on graph attention mechanism

Abstract

The invention discloses a contrast learning unsupervised cross-modal hash retrieval algorithm based on a graph attention mechanism, and relates to the technical field of cross-modal retrieval. According to the graph attention mechanism-based contrast learning unsupervised cross-modal Hash retrieval algorithm, feature extraction is performed on image and text data through a pre-trained CLIP model to obtain image modal features and text modal features, and additional positive samples are generated through data enhancement; the similarity is calculated through Euclidean distance and cosine similarity on the basis of a graph attention network, and high weight is given to positive samples and low weight is given to negative samples by comparing adjacent matrixes. The method has the advantages that the features of the image mode and the text mode can be effectively fused and mapped to the common Hash space, a graph attention mechanism can adaptively adjust the connection weight between the features of the different modes, samples with high semantic correlation are highlighted, and therefore the semantic consistency between the image and the text is ensured.

Description

Technical Field

[0001] The present invention relates to the field of cross-modal retrieval technology, and specifically to a contrastive learning unsupervised cross-modal hash retrieval algorithm based on a graph attention mechanism. Background Art

[0002] In recent years, with the rapid rise of social networking platforms, the volume and variety of multimedia content, including images, text, and video, have grown exponentially. According to statistics, Google alone processes more than 5 billion search queries every day, covering multiple forms of data such as text, images, videos, and audio. Therefore, effectively organizing and retrieving this large amount of information has become an urgent problem that needs to be solved. The essence of the cross-modal retrieval task is to input one type of data and retrieve another type of data, ensuring the semantic similarity between them as much as possible. For example, retrieving related images through text queries, or reversely retrieving text using image queries, such retrieval mechanisms can better meet the diverse needs of users, while improving the accuracy and efficiency of information search.

[0003] In order to deeply explore the deep semantic relationship between different modalities and achieve efficient cross-modal retrieval, researchers have proposed a variety of methods based on deep learning. For example, convolutional neural networks such as Alexnet or vgg16 are used to extract features, and different types of data are associated and matched by establishing a common feature representation space. In order to pursue efficient retrieval, feature values ​​are usually saved in the form of hash coding. In addition, depending on whether the input data of the training model has classification labels, the methods can be divided into two categories: supervised hashing and unsupervised hashing. Supervised hashing methods rely on labels for semantic judgment and can generate more discriminative hash codes, which usually have better results. However, in the face of today's massive Internet data, it is obviously impractical to manually label each training object. Unsupervised hashing methods only need to understand the associated semantic data, without the category of the object, so it has become a hot research direction.

[0004] Current unsupervised hashing methods usually use related cross-modal data to construct a similarity matrix, but due to the lack of appropriate references for comparison, the results are often not ideal. Although methods such as DJSRH and DGCPN attempt to optimize the similarity matrix and integrate semantic information within the modality, and have achieved certain results, there is still redundant information in the similarity comparison process, which makes the loss function unable to train a satisfactory model. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a contrastive learning unsupervised cross-modal hash retrieval algorithm based on a graph attention mechanism, which solves the problem that redundant information still exists in the similarity comparison process, resulting in the loss function being unable to train a satisfactory model.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a contrastive learning unsupervised cross-modal hash retrieval algorithm based on a graph attention mechanism, comprising the following steps: extracting features from image and text data through a pre-trained CLIP model to obtain image modal features and text modal features, and generating additional positive samples through data enhancement; calculating similarity through Euclidean distance and cosine similarity based on a graph attention network, assigning high weights to positive samples and low weights to negative samples by comparing adjacency matrices; mapping image modal features and text modal features to a common hash space through a graph attention network, generating hash codes for representing similar semantic features of multimodal data; using a loss function to adjust the distance between positive samples and negative samples, optimizing the generation of hash codes, and making the hash codes of positive samples closer and the hash codes of negative samples farther apart.

[0007] Furthermore, the feature extraction of image and text data is performed through the pre-trained CLIP model, and the specific analysis process is: obtaining an image data set and a text data set; using the CLIP model to encode the image data set and the text data set respectively, extracting feature representations, and obtaining image modal features and text modal features; normalizing the extracted image modal features and text modal features; and storing the normalized image modal features and text modal features in a feature library.

[0008] Furthermore, the additional positive samples are generated through data enhancement, and the specific analysis process is: select the original image samples and define the transformation strategy of data enhancement; apply random flipping and rotation geometric transformations to generate new image samples; add random noise to the generated new samples to enhance the diversity of the samples; combine the enhanced samples with the original samples to enrich the positive sample set.

[0009] Furthermore, by comparing the adjacency matrix, a high weight is assigned to the positive sample and a low weight is assigned to the negative sample. The specific analysis process is: calculate the similarity between samples, use Euclidean distance and cosine similarity to compare node pairs, and generate an adjacency matrix based on the similarity; in the adjacency matrix, a higher weight is assigned to samples from the same source, and a lower weight is assigned to mismatched samples.

[0010] Furthermore, the image modal features and text modal features are mapped to a common hash space through a graph attention network to generate a hash code. The specific analysis process is: the extracted image modal features and text modal features are respectively input into the graph attention network to establish connections between nodes; the image modal features and text modal features are fused through a multi-layer graph attention mechanism, and the features are grouped through an adaptive clustering method to obtain the final feature representation; the generated feature representation is processed through a nonlinear activation function, and the processed feature representation is linearly transformed to generate a fixed-length hash code.

[0011] Furthermore, the features are grouped by the adaptive clustering method, and the specific analysis process is: grouping the features by the adaptive clustering method to obtain the natural clustering pattern between the features; dynamically adjusting the number of clusters by the adaptive clustering method; using the clustering results as guiding information to optimize the hash code generation process and evaluate the impact of the clustering effect on the hash code generation quality.

[0012] Furthermore, the specific calculation process of the loss function is:

[0013]

[0014] In the formula, is the loss function, i is the index of the current sample, B i is the hash code of the i-th sample, B k is the hash code of the positive sample corresponding to the i-th sample, B j is the hash code of the negative sample in the batch, and τ is the temperature coefficient.

[0015] Furthermore, the generation of the optimized hash code is specifically analyzed as follows: calculating the contrast loss between the output of the current CLIP model and the target hash code to form a loss matrix to quantify the model performance; back-propagating the loss matrix using the gradient descent method, updating the CLIP model parameters using the momentum method, and setting the learning rate to adjust the amplitude of the CLIP model parameter update; regularly monitoring the training loss and validation loss of the CLIP model to determine the training effect of the CLIP model and prevent overfitting.

[0016] Furthermore, the setting of the learning rate regulates the amplitude of the CLIP model parameter update. The specific analysis process is: according to the performance of the CLIP model on the validation set, the initial learning rate and the learning rate decay strategy are set; after each training cycle, the validation loss is evaluated. If the validation loss does not decrease, the learning rate is reduced to improve the convergence of the model.

[0017] Furthermore, the momentum method is used to update the CLIP model parameters. The specific analysis process is as follows: a momentum term is maintained for each CLIP model parameter. When the CLIP model parameters are updated, the current gradient and momentum term are combined to calculate a new parameter value; the momentum coefficient is set to control the influence of the historical gradient on the current update; after each parameter update, the momentum term is recalculated and updated to reflect the new gradient information.

[0018] The present invention has the following beneficial effects:

[0019] (1) This contrastive learning unsupervised cross-modal hash retrieval algorithm based on graph attention mechanism can effectively fuse the features of image and text modalities and map them to a common hash space through the graph attention mechanism. The graph attention mechanism can adaptively adjust the connection weights between features of different modalities and highlight samples with strong semantic relevance, thereby ensuring the semantic consistency between images and texts. It solves the problem that traditional cross-modal retrieval methods are difficult to establish accurate semantic connections between different modalities (images and texts) due to the large differences in their feature expressions.

[0020] (2) This contrastive learning unsupervised cross-modal hash retrieval algorithm based on the graph attention mechanism can dynamically adjust the weights between samples so that the features related to the query task obtain higher weights, thereby enhancing the model's attention to important features. At the same time, through the multi-layer attention mechanism, the model can capture the complex nonlinear relationship between features and generate hash codes with more semantic expression capabilities, solving the problem that in cross-modal tasks, the semantic connection between samples is relatively complex and traditional linear models are difficult to capture such complex relationships.

[0021] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of the unsupervised cross-modal hash retrieval algorithm based on contrastive learning and graph attention mechanism of the present invention. DETAILED DESCRIPTION

[0023] The embodiment of the present application realizes an unsupervised cross-modal hash retrieval algorithm based on contrastive learning of a graph attention mechanism, and realizes that the features of image and text modalities can be effectively integrated through the graph attention mechanism and mapped to a common hash space. The graph attention mechanism can adaptively adjust the connection weights between features of different modalities, highlight samples with strong semantic relevance, thereby ensuring the advantage of semantic consistency between images and texts, and solving the problem that traditional cross-modal retrieval methods are difficult to establish accurate semantic connections between different modalities (images and texts) due to the large differences in their feature expressions.

[0024] See also Figure 1The embodiment of the present invention provides a technical solution: a contrastive learning unsupervised cross-modal hash retrieval algorithm based on a graph attention mechanism, comprising the following steps: extracting features of image and text data through a pre-trained CLIP model to obtain image modal features and text modal features, and generating additional positive samples through data enhancement; calculating similarity through Euclidean distance and cosine similarity based on a graph attention network, assigning high weights to positive samples and low weights to negative samples by comparing adjacency matrices; mapping image modal features and text modal features to a common hash space through a graph attention network, generating hash codes for representing similar semantic features of multimodal data; using a loss function to adjust the distance between positive samples and negative samples, optimizing the generation of hash codes, and making the hash codes of positive samples closer and the hash codes of negative samples farther away.

[0025] Specifically, the feature extraction of image and text data is performed through the pre-trained CLIP model, and the specific analysis process is: obtaining an image data set and a text data set; using the CLIP model to encode the image data set and the text data set respectively, extracting feature representations, and obtaining image modal features and text modal features; normalizing the extracted image modal features and text modal features; and storing the normalized image modal features and text modal features in a feature library.

[0026] In this embodiment, first, an image dataset and a text dataset are collected. The image dataset can include various types of pictures, such as product pictures, scene pictures, works of art, etc., and the text dataset is usually a descriptive text related to these pictures. The image dataset is usually composed of image files in formats such as JPEG and PNG. The images come from various sources, such as image libraries, social media, etc. The text dataset is usually a text description related to the image, which is a simple keyword, a complete sentence or a paragraph. The text description provides additional semantic information for the image to help the model learn the association between the image and the text. The CLIP model is a pre-trained model specifically used for cross-modal learning. By jointly training image and text data, the features of these two modalities can be effectively extracted. In this process, image encoding: The CLIP model uses a pre-trained convolutional neural network, such as ResNet or VisionTransformer, to encode the image dataset and extract the high-dimensional feature vector of the image. These feature vectors can capture the core visual information in the image and are expressed in numerical form. Text encoding: The CLIP model also uses a pre-trained language model, such as Transformer, to encode the text dataset and extract the semantic feature representation of the text. Similar to image modality features, text modality features are also saved in the form of high-dimensional vectors. The CLIP model can generate image modality features and text modality features. These feature vectors are usually located in the same feature space, ensuring that the semantic information of images and texts can be mapped and understood to each other. Feature normalization is a common operation in deep learning. The purpose is to map feature values ​​to a fixed range, such as between 0 and 1, so as to reduce the scale differences between different feature dimensions. Normalization can improve the convergence speed of the model and reduce the optimization problems caused by different numerical ranges. Image modality feature normalization: For the extracted image modality features, after normalization, each feature value will be in the same numerical range, so as to reduce the numerical differences between different image modality features and avoid the imbalance effect during training. Text modality feature normalization: Similarly, the normalization of text modality features ensures that the feature vectors are distributed in the same scale, which is convenient for subsequent calculation and comparison, especially in contrastive learning and hash code generation. After feature extraction and normalization, the modality features of image and text will be stored in the feature library. The feature library is the core of subsequent cross-modal retrieval, and the model will retrieve the features that best match the query from it. By mapping image and text data into the same feature space through the CLIP model, the semantic consistency of these modalities can be ensured, so that in cross-modal retrieval, such as querying images through text, or querying text through images, the semantic relationship between different modalities can be matched more accurately, thereby improving the accuracy of retrieval.

[0027] Specifically, the additional positive samples are generated through data enhancement, and the specific analysis process is: select the original image samples and define the transformation strategy of data enhancement; apply random flipping and rotation geometric transformations to generate new image samples; add random noise to the generated new samples to enhance the diversity of the samples; combine the enhanced samples with the original samples to enrich the positive sample set.

[0028] In this embodiment, first, select the original image samples for enhancement. The original image samples can come from the training set, representing the various features and patterns that the model needs to learn during the training process. Generally, the selected image samples should be highly representative and cover different categories and scenes to ensure that the generated enhanced samples are of practical value to the model training. Next, define the transformation strategy for data enhancement. The transformation strategy for data enhancement determines how to apply transformation operations to the original image, usually including geometric transformations such as flipping, rotation, scaling, and color transformations such as brightness and contrast adjustment. The definition of the transformation strategy should be designed according to the requirements of the task. For example, if the model needs to be robust to changes in the position of the object, the geometric transformation strategy will be more important; if it is necessary to handle changes in ambient lighting, color transformation may be more critical. Flip the image horizontally or vertically with a certain probability. This operation simulates the changes of the image at different viewing angles and is particularly suitable for object recognition and scene classification tasks. The image is randomly rotated within an angle range. The rotation operation helps the model adapt to the rotation invariance of the image, especially when the target may appear at different angles. Through the above geometric transformation, a variety of new samples can be generated from the original image. These new samples still maintain the semantic information of the original image, but enrich the data set by changing the angle and perspective. Based on the generated new samples, random noise is added to further enhance the diversity of the samples. Random noise usually refers to adding small random interference to the pixel values ​​of the image. This interference can simulate the noise scenes in practical applications, such as uneven light during shooting, camera jitter, or loss in data transmission. Noise type: Gaussian noise: Randomly perturb the pixel values ​​of the image, so that the image appears slightly blurred or changed in local areas, which can simulate the effect of sensor defects or signal interference during transmission. Salt and pepper noise: Randomly add white or black dots to the image to simulate image quality problems caused by low light environments or poor shooting equipment conditions. Other noises: Such as Poisson noise and uniform noise, can help the model maintain stability in more complex image environments. By adding noise, the model will encounter more challenging samples during training, thereby improving its ability to handle imperfect data in the real world. After geometric transformation and noise processing, the new enhanced samples are combined with the original samples to form an extended dataset.

[0029] Specifically, by comparing the adjacency matrix, high weights are assigned to positive samples and low weights are assigned to negative samples. The specific analysis process is: calculate the similarity between samples, use Euclidean distance and cosine similarity to compare node pairs, and generate an adjacency matrix based on the similarity; in the adjacency matrix, higher weights are assigned to samples from the same source, and lower weights are assigned to mismatched samples.

[0030] In this implementation scheme, this section describes the specific process of assigning high weights to positive samples and low weights to negative samples by comparing the adjacency matrix. The comparison of adjacency matrices is based on the similarity measurement between samples and is used to optimize the performance of the model in unsupervised learning. Similarity calculation is the basis for the model to distinguish positive samples from negative samples. The model first calculates the similarity between each pair of samples, an image and a text. The similarity calculation method includes: Euclidean distance: used to measure the spatial distance between the feature vectors of two samples. The smaller the distance, the more similar the samples are. Cosine similarity: used to evaluate the directional similarity between two samples. The closer to 1, the more consistent their feature directions. After the similarity calculation is completed, the model generates an adjacency matrix based on these similarities. If the similarity between the two samples is high, that is, the distance is close or the direction is consistent in the feature space, the corresponding elements in the adjacency matrix will be assigned a higher value. This means that the model believes that these samples are positive samples and should be paid more attention to in training. Conversely, if the similarity between the two samples is low, the corresponding elements in the matrix will be assigned a lower value or close to zero. It indicates that these samples are negative samples, and the model can reduce the attention to these sample pairs in training. For sample pairs identified as positive samples, an image and its corresponding description text, the model will give higher weights during the training process. Higher weights mean that these positive samples are more important in training, and the model will focus more on learning the feature relationships of these samples. This ensures that the model will prioritize learning semantic relationships with practical significance. Low weight assignment: For unmatched sample pairs, an image and an unrelated text, the similarity is low, and the model will assign lower weights to them or ignore them completely. The low weight setting can prevent negative samples from interfering with the model training process, thereby improving the efficiency of positive sample learning.

[0031] The comparison adjacency matrix details the and The degree of semantic relevance between them. The contrast adjacency matrix shows the semantic relevance between the image modality feature v1 and the text modality feature t1. Based on the pre-trained CLIP model, the original image modality feature H is extracted from the image and text respectively. v , text modality feature H t And contrast image modality feature H′ v, these features are mapped to the same encoding space through the pre-trained model, which greatly alleviates the heterogeneity problem between different modalities. The features of the three modalities are fused into a comprehensive feature matrix through splicing operations, and the calculation formula is:

[0032]

[0033] Where H is the integrated feature matrix after fusion, with a size of 3m×Z, m is the number of samples of each modality, Z is the feature dimension, and H v is the original image modality feature, H t is the text modality feature, H′ v To compare the image modality features.

[0034] It should be noted that by merging the features of the three modalities, namely the original image modal features, the text modal features and the contrast image modal features, into a matrix, the model can map data of different modalities into the same encoding space for processing, which can reduce the modal differences between images and texts and help capture the semantic relevance between the two.

[0035] The model measures the similarity between samples by constructing an affinity matrix. The similarity consists of two parts: cosine similarity and Euclidean distance similarity. Cosine similarity calculates the angular similarity of two feature vectors, reflecting whether the directions of the two feature vectors are consistent. A higher cosine similarity indicates that the directions of the samples in the feature space are similar. Euclidean distance similarity is based on the distance measurement between feature vectors and is used to calculate the spatial distance between two samples. The smaller the distance, the more similar the samples are. The comprehensive calculation formula for similarity measurement is:

[0036]

[0037] In the formula, S ij is the affinity matrix, C ij is the cosine similarity, D ij is the Euclidean distance similarity, and is the normalized feature, and ρ is the scaling parameter used to control the influence range of the Euclidean distance.

[0038] By combining cosine similarity and Euclidean distance similarity, the model is able to more comprehensively evaluate the similarity between samples, considering not only the angular relationship between sample feature vectors but also their absolute distance in space. This approach can more accurately reveal the potential semantic associations between samples.

[0039] After obtaining the similarity matrix, the calculation formula for the alignment adjacency matrix is ​​as follows:

[0040]

[0041] In the formula, A ij is the adjacency matrix, i and j are samples, when the semantic relationship between samples i and j is very close, that is, a positive sample pair, such as an image and text pair, then the highest weight A is directly assigned to them. ij = 1, indicating a strong connection between them. For incompletely matched sample pairs, i.e. non-positive samples, their connection weight depends on the similarity matrix S ij If S ij If it exceeds a certain threshold ∈, then according to ηS ij Assign lower weights to them, indicating their weak connection. When the similarity between sample pairs is insufficient, that is, ηS ij If it is less than the threshold ∈, it is considered that there is no correlation between them and the weight is set to 0.

[0042] Specifically, the image modal features and text modal features are mapped to a common hash space through a graph attention network to generate a hash code. The specific analysis process is: the extracted image modal features and text modal features are respectively input into the graph attention network to establish connections between nodes; the image modal features and text modal features are fused through a multi-layer graph attention mechanism, and the features are grouped through an adaptive clustering method to obtain the final feature representation; the generated feature representation is processed through a nonlinear activation function, and the processed feature representation is transformed linearly to generate a fixed-length hash code.

[0043] In this embodiment, first, the model extracts high-dimensional features of images and texts respectively through the pre-trained CLIP model, and the feature vectors of each image and text represent the semantic information of the sample in the feature space. Next, these image modality features and text modality features are input into the graph attention network. In this network, the feature vector of each sample is regarded as a node in the graph, and the connection between the nodes represents their similarity or relevance. By calculating the similarity between pairs of nodes, such as cosine similarity or Euclidean distance, the model establishes connection edges for each pair of nodes, which can be strong connections (positive samples) or weak connections (negative samples), and the weights of different connections will be adjusted in subsequent steps. The core of the graph attention network is that it can dynamically adjust the weights between different nodes. Through the multi-layer attention mechanism, the model can learn the relationship between samples layer by layer and assign higher weights to more important features. At each layer, the model calculates the attention weights between each pair of nodes. The higher the weight, the closer the semantic relationship between these nodes. The model gradually learns the potential connection between them by fusing image modality features and text modality features. After fusing the image modality features and text modality features, the model uses an adaptive clustering method to group the features. Specifically, adaptive clustering dynamically determines the number and form of clusters based on the similarity and distribution of features. In this way, the model can automatically discover semantic clusters between different samples and further optimize the generation of hash codes based on these clusters. After obtaining the final feature representation, the model applies nonlinear activation functions such as ReLU or tanh to the image modality features and text modality features. Nonlinear activation functions can introduce nonlinear factors, increase the expressiveness of features, and enable the model to handle more complex semantic information. The features after nonlinear activation are further mapped to a fixed-length hash space through linear transformation. The hash code is a binary representation generated by linear transformation, which is used to quickly compare the similarity of samples in subsequent retrieval tasks.

[0044] Specifically, the features are grouped by the adaptive clustering method, and the specific analysis process is: grouping the features by the adaptive clustering method to obtain the natural clustering pattern between the features; dynamically adjusting the number of clusters by the adaptive clustering method; using the clustering results as guiding information to optimize the hash code generation process and evaluate the impact of the clustering effect on the hash code generation quality.

[0045] In this embodiment, the adaptive clustering method refers to automatically grouping features according to the similarity and distribution characteristics between features. In this process, the model analyzes each sample feature and identifies the natural clustering pattern between them. Generally speaking, similar sample features will be clustered in the same cluster, which can help the model identify the potential semantic relationship in the feature space. Different features, such as image features and text features, form multiple clusters according to similarity distribution, and the features in each cluster are semantically closer. Through this adaptive clustering, the model can find the semantic distribution law between features, so as to better organize and represent these data. Dynamically adjust the number of clusters. Different from the traditional fixed clustering method, adaptive clustering allows the model to dynamically adjust the number of clusters according to the complexity and distribution of the features. The model will select the appropriate number of clusters according to the feature structure of the data set without presetting a fixed number of clusters. If the data distribution is sparse and complex, the model will automatically increase the number of clusters to more finely describe the relationship between features; if the data distribution is relatively concentrated and uniform, the model will reduce the number of clusters to avoid over-segmentation of samples and ensure that the clustering results are concise and clear. After clustering is completed, the generation of hash codes is optimized according to the clustering results. Each cluster represents similar features in specific semantics. These features should be mapped to similar positions in the hash space. Clustering ensures that samples in the same cluster have similar hash codes, while the hash codes of samples in different clusters maintain large differences. After generating hash codes, the impact of clustering results on the quality of hash code generation is evaluated. By calculating indicators such as retrieval accuracy and recall rate, it is evaluated whether the hash codes accurately represent the semantic relationship between samples. If the clustering effect is not ideal, the number of clusters or parameters can be further adjusted to improve the quality of hash code generation. The adaptive clustering method provides the flexibility to dynamically adjust the number of clusters and can discover semantic associations between features based on the natural distribution law of features.

[0046] Specifically, the specific calculation process of the loss function is:

[0047]

[0048] In the formula, is the loss function, i is the index of the current sample, B i is the hash code of the i-th sample, B k is the hash code of the positive sample corresponding to the i-th sample, B j is the hash code of the negative sample in the batch, and τ is the temperature coefficient.

[0049] In this implementation, the loss function is used to measure the loss value of the model's effectiveness in distinguishing positive samples from negative samples. By minimizing this loss, the model can gradually learn to reduce the distance between the hash codes of similar positive samples and increase the distance between the hash codes of dissimilar negative samples. The index of the current sample ranges from 0 to m-1, where m is the total number of samples in a batch. This index is used to mark the hash code of each sample. B i is the hash code of the i-th sample. The hash code is a binary vector used to represent the position of the sample in the hash space. By optimizing the loss function, the model adjusts these hash codes so that similar samples generate similar hash codes and dissimilar samples generate hash codes with greater differences. k is the hash code of the positive sample corresponding to the i-th sample. For example, if the i-th sample is an image, B k is the hash code of the text corresponding to the image. This partial loss function ensures that the hash codes of positive samples are closer in the hash space. j is the hash code of the negative samples in the batch. Negative samples refer to those samples that have no direct relationship with the current sample. The loss function reduces the similarity between these negative samples and the hash code of the current sample, thereby increasing the distance between them, so that the model can better distinguish between positive and negative samples. τ is the temperature coefficient used to control the model's sensitivity to distinguishing between positive and negative samples. A lower τ value makes the model more sensitive to distinguishing between positive and negative samples, while a higher τ value reduces the model's ability to distinguish. In the formula, the numerator represents the similarity between the current sample and its corresponding positive sample. Their similarity is calculated by the dot product of the hash codes to obtain a similarity score, which is then amplified by an exponential function to enhance the model's learning effect on positive samples. The denominator represents the sum of the similarities between the current sample and all other samples in the batch, including positive and negative samples. The model compares the current sample with all samples in the batch and tries to distance the positive and negative samples as much as possible. The summation symbol in the loss function indicates that the loss is calculated for each sample and added up as the total loss for the entire batch. The model optimizes the generation of hash codes by minimizing the total loss, making the hash codes of positive samples close and the hash codes of negative samples far apart. By comparing the loss function, the similarities between positive and negative samples can be effectively distinguished.

[0050] Specifically, the generation of the optimized hash code includes the following specific analysis processes: calculating the contrast loss between the output of the current CLIP model and the target hash code to form a loss matrix to quantify the model performance; back-propagating the loss matrix using the gradient descent method, updating the CLIP model parameters using the momentum method, and setting the learning rate to adjust the amplitude of the CLIP model parameter update; and regularly monitoring the training loss and validation loss of the CLIP model to determine the training effect of the CLIP model and prevent overfitting.

[0051] In this embodiment, first, the model extracts the features of the image and text through CLIP to generate a hash code for each sample. In order to optimize the generation of these hash codes, it is necessary to calculate the contrast loss between the currently generated hash code and the target hash code. This loss function measures the performance of the model in generating hash codes of similar samples, that is, whether the hash code of the positive sample is close enough and whether the hash code of the negative sample is far enough. The greater the similarity between the positive samples and the smaller the similarity between the negative samples, the smaller the loss. By minimizing this contrast loss, a more effective hash code can be generated. The loss matrix is ​​formed based on the contrast loss calculation of each sample and is used to quantify the performance of the model. Each element in the matrix corresponds to the loss value of a sample pair, and the total loss is calculated by accumulating the losses of all samples. After calculating the loss, the model back-propagates the loss matrix through the gradient descent method to calculate the gradient of each model parameter relative to the loss function. Through back-propagation, it is possible to understand which parameters have a greater impact on the current loss, and reduce the loss value by adjusting these parameters. During the gradient descent process, the model uses the momentum method to update the parameters of the CLIP model. Momentum method is a variant of gradient descent, which accelerates the movement of the model in the direction of gradient descent by introducing a momentum term. It can smooth the parameter update process, prevent the model from oscillating near the local optimal point, and accelerate convergence. The learning rate is a key hyperparameter that controls the amplitude of parameter updates. During the optimization process, the model will determine the size of each parameter update based on the set learning rate. If the learning rate is too large, the model is likely to skip the optimal solution; if the learning rate is too small, the model converges slowly. The learning rate is usually adjusted dynamically according to the performance of the model to ensure that the parameter update process is efficient and stable. During the training process of the model, it is necessary to continuously monitor the loss value on the training set to ensure that the model is continuously optimized. The training loss value should gradually decrease as the training process progresses, indicating that the model is learning the characteristics of the input data. In addition to the training loss, it is also necessary to regularly monitor the loss on the validation set. The data of the validation set is not used for model training, and it can reflect the performance of the model on unseen data. Therefore, the change in validation loss can judge the generalization ability of the model. If the validation loss starts to rise while the training loss continues to decrease, it indicates that the model may be overfitting, and training should be stopped early to prevent the model from overfitting the training set and failing to perform well on new data. Through the optimization process, the model can effectively generate more accurate hash codes.

[0052] Specifically, the setting of the learning rate adjusts the amplitude of the CLIP model parameter update. The specific analysis process is: according to the performance of the CLIP model on the validation set, the initial learning rate and the learning rate decay strategy are set; after each training cycle, the validation loss is evaluated. If the validation loss does not decrease, the learning rate is reduced to improve the convergence of the model.

[0053] In this embodiment, the initial learning rate determines the parameter update amplitude of the model in the early stage of training. A higher initial learning rate can accelerate the training of the model, while a lower initial learning rate makes the model more robust. Usually, the initial learning rate is set by experiment or selected based on the initial performance of the model on the validation set. The setting of the initial learning rate needs to be weighed according to the complexity of the task, the scale of the model, and the diversity of the data set. A larger initial learning rate: suitable for the early stage of model training. When the model is far from the optimal solution, a larger parameter update can accelerate learning. A smaller initial learning rate: in the later stage of training, when the model approaches the optimal solution, a smaller learning rate can adjust the parameters in detail to avoid skipping the local optimal point. As the training proceeds, a fixed learning rate may not be able to adapt to the different stages of the model. Therefore, adopting a learning rate decay strategy is an important means to improve the model training effect. As the training deepens, the learning rate is gradually reduced so that the model can adjust the parameters more accurately during the later training. Common learning rate decay methods include exponential decay (the learning rate is gradually reduced at an exponential level), step decay (reducing the learning rate after a specific training round), and adaptive decay based on model performance (such as triggering decay when the verification loss no longer decreases). After each training cycle, the performance of the model should be evaluated not only on the training set, but also on the validation set. The loss on the validation set can reflect the performance of the model on unseen data and help detect whether overfitting has occurred. If the validation loss decreases, it means that the model is learning effectively during training; if the validation loss stagnates or increases, it means that the model may be overfitting and the training strategy needs to be adjusted. The validation loss gradually decreases in consecutive training cycles, indicating that the current learning rate is set reasonably and the model is converging. In this case, just keep the current learning rate. If the validation loss does not decrease or starts to increase, it means that the current learning rate may be too large, causing the model to fail to converge stably. At this point, the model should automatically reduce the learning rate. A smaller learning rate can reduce the magnitude of each parameter update, allowing the model to optimize parameters in a smaller step size, ensuring that the model gradually approaches the optimal solution. By dynamically adjusting the learning rate, the model can flexibly adapt to the needs of different training stages.

[0054] Specifically, the momentum method is used to update the CLIP model parameters. The specific analysis process is as follows: a momentum term is maintained for each CLIP model parameter. When the CLIP model parameters are updated, the current gradient and momentum term are combined to calculate the new parameter value; the momentum coefficient is set to control the influence of the historical gradient on the current update; after each parameter update, the momentum term is recalculated and updated to reflect the new gradient information.

[0055] In this embodiment, each time the parameters of the CLIP model are updated, in addition to using the current gradient information, a momentum term needs to be maintained for each parameter. The momentum term records the cumulative gradient information in the previous rounds of parameter updates, and the purpose is to use the direction of the historical gradient to accelerate the update of the parameters in the direction of the optimal solution. Each time the parameters of the CLIP model are updated, it not only depends on the current gradient, but also combines the momentum terms of the previous rounds (i.e., the cumulative effect of the historical gradient). This combination can help the model maintain the stability of the update when the gradient changes greatly, so that the model can continue to move forward in the same direction instead of frequently changing the update direction. This helps to speed up the convergence of the model. A key part of the momentum method is the momentum coefficient, which determines the degree of influence of the historical gradient in the current parameter update. The momentum coefficient is usually set to a value close to 1, such as 0.9 or 0.99, which means that 90% or 99% of the momentum term information will be retained each time the update is made, while only a small part of the current gradient information will be retained. By setting the momentum coefficient, the model can find a balance between the current gradient and the historical gradient. If the momentum coefficient is high, the model will rely more on historical gradients, making the parameter update direction smoother and avoiding excessive oscillations in complex loss surfaces. If the momentum coefficient is low, the model will rely more on the current gradient, which is suitable for situations where the gradient changes are small and the loss surface is smoother. After each parameter update, the model will recalculate the momentum term to reflect the latest information of the current gradient. The momentum term not only accumulates previous gradient changes, but also takes into account the latest gradient update direction, which can better guide the model in the direction of the optimal solution. By using the momentum method to update the parameters of the CLIP model, the model can significantly speed up the convergence speed while maintaining the stability of the update process.

[0056] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] The present invention is described with reference to flowcharts and / or block diagrams of systems, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0058] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0059] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0060] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0061] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. Unsupervised cross-modal hash retrieval algorithm based on contrastive learning of graph attention mechanism, characterized by: The following steps are involved: The pre-trained CLIP model is used to extract features from image and text data to obtain image modality features and text modality features, and additional positive samples are generated through data enhancement. Based on the graph attention network, similarity is calculated through Euclidean distance and cosine similarity. By comparing the adjacency matrix, high weights are given to positive samples and low weights are given to negative samples. The image modality features and text modality features are mapped to a common hash space through a graph attention network to generate hash codes to represent similar semantic features of multimodal data; The loss function is used to adjust the distance between positive samples and negative samples and optimize the generation of hash codes, making the hash codes of positive samples closer and the hash codes of negative samples farther away.

2. The unsupervised cross-modal hash retrieval algorithm based on contrastive learning and graph attention mechanism according to claim 1 is characterized in that: The pre-trained CLIP model is used to extract features from image and text data. The specific analysis process is as follows: Get image dataset and text dataset; The CLIP model is used to encode the image dataset and the text dataset respectively, extract feature representations, and obtain image modality features and text modality features; Normalize the extracted image modality features and text modality features; The normalized image modal features and text modal features are stored in the feature library.

3. The unsupervised cross-modal hash retrieval algorithm based on contrastive learning and graph attention mechanism according to claim 1 is characterized in that: The additional positive samples are generated by data enhancement. The specific analysis process is as follows: Select original image samples and define transformation strategies for data augmentation; Apply random flipping and rotation geometric transformations to generate new image samples; Add random noise to the generated new samples to enhance the diversity of samples; Combine the enhanced samples with the original samples to enrich the positive sample set.

4. The unsupervised cross-modal hash retrieval algorithm based on contrastive learning and graph attention mechanism according to claim 1, characterized in that: By comparing the adjacency matrix, high weights are assigned to positive samples and low weights are assigned to negative samples. The specific analysis process is as follows: Calculate the similarity between samples, use Euclidean distance and cosine similarity to compare node pairs, and generate an adjacency matrix based on the similarity; In the adjacency matrix, higher weights are given to samples from the same source, and lower weights are given to mismatched samples.

5. The unsupervised cross-modal hash retrieval algorithm based on contrastive learning and graph attention mechanism according to claim 1, characterized in that: The image modality features and text modality features are mapped to a common hash space through the graph attention network to generate hash codes. The specific analysis process is as follows: The extracted image modality features and text modality features are respectively input into the graph attention network to establish connections between nodes; Through the multi-layer graph attention mechanism, image modality features and text modality features are fused, and the features are grouped through the adaptive clustering method to obtain the final feature representation; The generated feature representation is processed through a nonlinear activation function, and the processed feature representation is transformed linearly to generate a hash code of fixed length.

6. The unsupervised cross-modal hash retrieval algorithm based on contrastive learning and graph attention mechanism according to claim 5 is characterized in that: The features are grouped by the adaptive clustering method, and the specific analysis process is as follows: The features are grouped by adaptive clustering methods to obtain natural clustering patterns between features; Dynamically adjust the number of clusters through adaptive clustering methods; The clustering results are used as guiding information to optimize the hash code generation process and evaluate the impact of clustering effect on the quality of hash code generation.

7. The unsupervised cross-modal hash retrieval algorithm based on contrastive learning and graph attention mechanism according to claim 1, characterized in that: The specific calculation process of the loss function is: In the formula, is the loss function, i is the index of the current sample, B i is the hash code of the i-th sample, B k is the hash code of the positive sample corresponding to the i-th sample, B j is the hash code of the negative sample in the batch, and τ is the temperature coefficient.

8. The unsupervised cross-modal hash retrieval algorithm based on contrastive learning and graph attention mechanism according to claim 1, characterized in that: The generation of the optimized hash code is specifically analyzed as follows: Calculate the contrast loss between the output of the current CLIP model and the target hash code to form a loss matrix to quantify the model performance; Use the gradient descent method to back-propagate the loss matrix, use the momentum method to update the CLIP model parameters, and set the learning rate to adjust the amplitude of the CLIP model parameter update; Regularly monitor the training loss and validation loss of the CLIP model to judge the training effect of the CLIP model and prevent overfitting.

9. The unsupervised cross-modal hash retrieval algorithm based on contrastive learning and graph attention mechanism according to claim 8, characterized in that: The setting of the learning rate adjusts the amplitude of the CLIP model parameter update. The specific analysis process is as follows: According to the performance of the CLIP model on the validation set, set the initial learning rate and learning rate decay strategy; After each training cycle, the validation loss is evaluated. If the validation loss does not decrease, the learning rate is reduced to improve the convergence of the model.

10. The unsupervised cross-modal hash retrieval algorithm based on contrastive learning and graph attention mechanism according to claim 9, characterized in that: The momentum method is used to update the CLIP model parameters. The specific analysis process is as follows: A momentum term is maintained for each CLIP model parameter. When the CLIP model parameter is updated, the current gradient and momentum term are combined to calculate the new parameter value. Setting the momentum coefficient controls the influence of historical gradients on the current update; After each parameter update, the momentum term is recalculated and updated to reflect the new gradient information.

Images