Semantic fusion deep hash image retrieval method

Abstract

The invention discloses a semantic fusion deep hash image retrieval method, relates to the technical field of data processing, and aims at generating a semantic similarity matrix by combining label level and semantic level similarity, considering category similarity and high-level semantic similarity and adapting to a multi-label image retrieval scene. And the convolutional neural network based on correlation operation extracts intra-layer self-correlation and inter-layer cross-correlation features, mines high-order statistical information of the image, makes up for the defect of insufficient first-order feature discrimination, and improves the Hash coding quality. The overall cost function is constructed in combination with multi-similarity loss and binary quantization loss, the influence of sample imbalance on model training is effectively relieved, model learning is made to have discriminative Hash coding, rapid and efficient image retrieval is finally achieved, the retrieval speed and precision are considered, and the requirement for massive image retrieval in the scene of the Internet of Things can be better met. Technical pain points existing in an existing deep supervision Hash method are effectively solved, and the efficiency, precision and applicability of image retrieval are improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a semantic fusion-based deep hash image retrieval method. Background Technology

[0002] With the rapid development of IoT technology, the multimedia data collected by various image sensing devices is growing exponentially. Quickly and efficiently retrieving target images from massive image databases has become a research hotspot, and hash learning, as a representative method in the field of massive data retrieval, has experienced rapid development. Hash functions can encode the high-dimensional features of an image into a compact binary hash code while maintaining the semantic similarity between the original images. Existing hash methods are mainly divided into three categories: supervised, semi-supervised, and unsupervised. Among them, supervised hash learning, guided by manual annotation, can generate more discriminative binary codes and has significant advantages in retrieval performance.

[0003] Breakthroughs in deep neural networks for visual tasks such as image classification, object recognition, and semantic segmentation have driven the development of deep supervised hashing methods. Researchers have integrated feature extraction and hash code learning into a single framework, proposing a deep learning-based hashing framework. This framework has achieved excellent retrieval performance on relevant benchmark datasets and has been widely applied in IoT image retrieval engine services. Compared to traditional hashing methods, deep supervised hashing, through end-to-end model design, integrates feature extraction and binary code learning, which is more conducive to generating binary codes with high-order similarity, further improving the efficiency and effectiveness of image retrieval.

[0004] Despite significant performance improvements in deep supervised hashing methods, several technical challenges remain in practical applications: First, most image hashing methods measure image pair similarity in a coarse-grained manner. Even those methods that define similarity by the number of shared labels only represent category similarity, neglecting the high-level semantic similarity between image pairs, making them unsuitable for multi-label image retrieval. Second, the training process of existing hashing models is susceptible to imbalanced samples, causing the model to favor learning the category with a larger number of samples, significantly reducing retrieval performance. Third, most deep hashing models only use first-order image features for hash learning, ignoring the effectiveness of higher-order statistical information in hash learning. This results in insufficient discriminative power of the extracted image features, limiting the quality of hash encoding and the overall retrieval performance of the model. Summary of the Invention

[0005] In order to solve the above-mentioned technical problems, this application proposes the following technical solution: In a first aspect, embodiments of this application provide a semantic fusion-based deep hash image retrieval method, including: Obtain an image dataset with class labels and initialize a depth hashing model, then adjust the input image size according to the initial depth hashing model; The combined label-level and semantic-level similarity of images generates a semantic similarity matrix representing fine-grained differences between the images; Convolutional neural networks based on correlation operations extract intra-layer autocorrelation features and inter-layer cross-correlation features from the depth features of images, generating high-order feature vectors containing high-order statistical information. Using the higher-order feature vector as input, a multi-similarity loss is constructed by combining the semantic similarity matrix, and the overall cost function is formed by combining it with the binary quantization loss. This is used to train a deep hash model and learn hash encoding. Generate the hash code corresponding to the query image, and generate a list of search results based on the hash codes of the query image and the candidate images to complete the image retrieval.

[0006] In one possible implementation, the step of acquiring the image dataset containing class labels and initializing a depth hashing model, and adjusting the input image size according to the initial depth hashing model, includes: Determine a training set containing N original images. and a corresponding tag set containing C categories ; The deep hashing model was initialized using AlexNet pre-trained on ImageNet as the backbone CNN network; Adjust all images to the input size suitable for the backbone CNN network, and complete the partitioning of the dataset into training, testing, and query databases.

[0007] In one possible implementation, the joint image's label-level similarity and semantic-level similarity generate a semantic similarity matrix representing fine-grained differences between image pairs, including: Extract each pair of images from the training set. and tag set and and determine the and semantic feature vector and ; by and Intersection and Union As a label-level similarity, and with and cosine similarity As a semantic-level similarity; The semantic similarity of image pairs is obtained by fusing the label-level similarity and semantic-level similarity. : in: As the core quantitative indicator of fusion-type similarity, These are the weighting coefficients. express and The inner product, The norm of a vector; Iterate through all image pairs to construct a semantic similarity matrix. .

[0008] In one possible implementation, the joint image's label-level similarity and semantic-level similarity generate a semantic similarity matrix representing fine-grained differences between image pairs, including: Extract each pair of images from the training set. and tag set and and determine the and semantic feature vector and ; by and Intersection and Union As a label-level similarity, and with and cosine similarity As a semantic-level similarity; The semantic similarity of image pairs is obtained by fusing the label-level similarity and semantic-level similarity. : in: As the core quantitative indicator of fusion-type similarity, These are the weighting coefficients. express and The inner product, The norm of a vector; Iterate through all image pairs to construct a semantic similarity matrix. .

[0009] In one possible implementation, higher-order statistical learning optimizes parameters through backpropagation, and the feature matrix corresponding to the intra-layer autocorrelation is obtained. The partial derivatives are calculated as follows: Feature matrix corresponding to inter-layer cross-correlation features and The partial derivatives are calculated as follows: in: This is the global cost function.

[0010] In one possible implementation, the step of using the higher-order feature vector as input, constructing a multi-similarity loss by combining the semantic similarity matrix, and combining this with the overall cost function consisting of binary quantization loss, to train a deep hash model and learn hash encoding includes: Combine semantic similarity matrix to filter valid positive and negative sample pairs; Define self-similarity and negative relative similarity, calculate the weights for positive and negative sample pairs respectively, and implement differentiated weighting for different sample pairs; In multiple similarity loss functions Semantic similarity is introduced to optimize the similarity of sample pairs. The partial derivatives are used to calculate the weights for positive and negative sample pairs; Add regularized binary quantization loss The L1 norm-constrained hash encoding output is close to standard binary form: Where: 1 represents the identity matrix, Representing vectors L 1-norm, It is the absolute value of the element. and The feature vector output by the hash stream; By introducing weighting coefficients to balance the effects of multi-similarity loss and quantization loss, the overall cost function is obtained through fusion: Where: λ is the weighting coefficient that controls the quantization error.

[0011] In one possible implementation, the combination of semantic similarity matrix Filtering valid positive and negative sample pairs includes: Based on the relationship between Hamming distance and inner product, the inner product is used to define the similarity of image pairs: ,in: and Image pairs and binary encoding; Combined with semantic similarity To obtain the optimal similarity between image pairs: ; Set threshold Filter positive and negative sample pairs. Positive sample pairs must meet the following conditions: Negative sample pairs must meet the following conditions: in: To maximize the calculated result for all semantically similar images, This indicates that the minimum value is taken for all semantically dissimilar images.

[0012] In one possible implementation, the definition of self-similarity and negative relative similarity, and the calculation of weights for positive and negative sample pairs respectively, to achieve differentiated weighting for different sample pairs, includes: By directly calculating pairwise similarity and negative sample pairs Difference defines self-similarity and negative relative similarity ; For negative sample pairs The weights for negative sample pairs are calculated as follows: For positive sample pairs The weights for positive sample pairs are calculated as follows: in: For For the set of all negative sample pairs of the anchor sample, For For the set of all positive sample pairs of the anchor sample, The scaling factor hyperparameter is used to calculate the weights of positive sample pairs. and The threshold hyperparameter is calculated for the weights of negative sample pairs.

[0013] In one possible implementation, the for: .

[0014] In one possible implementation, generating the hash code corresponding to the query image, and generating a search result list based on the query image hash code and candidate image hash codes to complete the image retrieval, includes: The deep hash model is used to generate the binary hash code corresponding to the query image; Calculate the Hamming distance between the query image hash code and the candidate image hash codes: in: and Image pairs and binary encoding, The length of the binary hash code; Candidate images are sorted by Hamming distance from smallest to largest to generate a list of search results, thus completing the image retrieval.

[0015] In this embodiment, a semantic similarity matrix is ​​generated by jointly using label-level and semantic-level similarity, overcoming the limitations of traditional coarse-grained similarity measures. This approach considers both category similarity and high-level semantic similarity, adapting to multi-label image retrieval scenarios and addressing the problem of existing methods ignoring fine-grained semantic differences. A convolutional neural network based on correlation operations extracts intra-layer autocorrelation and inter-layer cross-correlation features, mining higher-order statistical information of the image and compensating for the insufficient discriminative power of first-order features, thus improving the quality of hash encoding. A comprehensive cost function is constructed by combining multi-similarity loss and binary quantization loss, effectively mitigating the impact of sample imbalance on model training. This enables the model to learn more discriminative hash encoding, ultimately achieving fast and efficient image retrieval, balancing retrieval speed and accuracy, and better adapting to the needs of massive image retrieval in IoT scenarios. This effectively solves the technical pain points of existing deep supervised hashing methods, improving the efficiency, accuracy, and applicability of image retrieval. Attached Figure Description

[0016] Figure 1 A schematic diagram of a semantic fusion-based deep hash image retrieval method provided in this application embodiment; Figure 2 This is a schematic diagram illustrating the generation of a semantic similarity matrix provided in an embodiment of this application; Figure 3 This is a schematic diagram of a traditional bilinear convolutional neural network; Figure 4 A schematic diagram of bilinear pooling for cross-layer features provided in this application embodiment; Figure 5 A schematic diagram illustrating hash encoding learning provided in an embodiment of this application; Figure 6 This is a schematic diagram of the P@H≤2 curve provided in the embodiments of this application; Figure 7 This is a schematic diagram of the TopN accuracy curve provided in an embodiment of this application; Figure 8 A schematic diagram of the accuracy-recall curve provided for embodiments of this application; Figure 9 The different datasets provided in this application embodiment on the MS-COCO dataset and WeightedmAP score; Figure 10 Different on the NUS-WIDE dataset provided for embodiments of this application and WeightedmAP score; Figure 11 Different datasets on the MIRFLICKR-25K dataset provided for embodiments of this application and WeightedmAP score; Figure 12 The different datasets provided in this application embodiment on the MS-COCO dataset WeightedmAP score; Figure 13 Different on the NUS-WIDE dataset provided for embodiments of this application WeightedmAP score; Figure 14 Different datasets on the MIRFLICKR-25K dataset provided for embodiments of this application WeightedmAP score; Figure 15 The different datasets provided in this application embodiment on the MS-COCO dataset WeightedmAP score; Figure 16 Different on the NUS-WIDE dataset provided for embodiments of this application WeightedmAP score; Figure 17 Different datasets on the MIRFLICKR-25K dataset provided for embodiments of this application WeightedmAP score. Detailed Implementation

[0017] The present solution will now be described in conjunction with the accompanying drawings and specific embodiments.

[0018] See Figure 1 The semantic fusion-based deep hashing image retrieval method provided in this embodiment includes: S101, Obtain an image dataset containing class labels and initialize a depth hash model, and adjust the input image size according to the initial depth hash model.

[0019] This embodiment first determines a training set containing N original images. and a corresponding tag set containing C categories For the deep hashing model, a pre-trained CNN network is used to initialize the deep hashing model on ImageNet, and a pre-trained AlexNet is used as the backbone network to obtain fine-grained features.

[0020] After determining the deep hashing model, all images are adjusted to the input size suitable for the backbone CNN network, completing the partitioning of the dataset into training, testing, and query databases. In this embodiment, the image sizes of the three datasets are adjusted to... This ensures that the data meets the input size of the underlying network.

[0021] S102, Combine the label-level similarity and semantic-level similarity of the images to generate a semantic similarity matrix that represents the fine-grained differences between the image pairs.

[0022] Some multi-label hashing methods roughly define the similarity between image pairs based on the number of shared labels. Specifically, if images and If at least one tag is shared, then If the image and If tags are not shared, then This definition pattern can only simply describe tag-level similarity, ignoring high-level semantic similarity and limiting further improvements in retrieval accuracy. To address this issue, this embodiment designs a semantic similarity metric by jointly learning class-tag-based similarity and semantic-based similarity, such as... Figure 2 As shown, semantic similarity is defined. : The expression is: in: and These represent the original images. and The tag set. As the weighting coefficient, based on experience, Set to 0.3; and It is a 512-dimensional semantic feature vector, i.e., a high-level semantic feature descriptor, which is obtained by exploring the original image through a general sentence encoder. and The relevant text is encoded and generated. Manually annotated text information mainly refers to high-level semantic descriptions, such as scene descriptions and interaction information between different objects. express and The inner product, The norm of a vector.

[0023] The similarity of image pairs integrates label-level similarity and semantic-level similarity. Based on the above quantization method, It can represent high-level semantic similarity, capable of distinguishing differences beyond class label information. Finally, it iterates through all image pairs to construct a semantic similarity matrix. .

[0024] S103, a convolutional neural network based on correlation operations, extracts intra-layer autocorrelation features and inter-layer cross-correlation features from the deep features of an image, generating a high-order feature vector containing high-order statistical information.

[0025] Feature learning is a crucial step in hash learning, and the ability of the designed hash model to extract deep semantic features directly impacts the discriminativeness of the generated hash codes. Therefore, deep hash networks often exhibit better performance than traditional hashing methods. To obtain high-order semantic features, previous studies have mostly adopted bilinear convolutional neural networks (B-CNNs) as their basic architecture and generated discriminative hash codes through hash learning. As shown in Figure 3, B-CNNs typically employ two independent CNN models, each learning feature descriptors from the input image. However, the B-CNN model only captures the cross-correlation features of the cross-convolutions, neglecting the autocorrelation features of a single convolutional layer. Furthermore, previous research has demonstrated the important role of autocorrelation features in generating discriminative feature descriptors.

[0026] In the feature learning of this embodiment, two types of correlation operations are used to obtain cross-correlation and autocorrelation information, thereby generating image features with high-level statistical vectors. For example... Figure 4 As shown below, the learning of higher-order statistics will be explained in detail.

[0027] The semantic features of the input image are obtained using two independent CNN models, CNNI and CNNII, to generate feature matrices X and Y, which are then input into a bilinear pooling module to generate higher-order features Z. in , represents the feature matrices generated by CNNI and CNNII, respectively. n and d represent the number of features and their dimension, respectively. The feature extraction proposed in this embodiment contains only one feature stream, similar to B-CNN, with intra-layer autocorrelation features. for: The parameters were optimized using the backpropagation algorithm, and the partial derivatives of the objective function were obtained. The feature matrix corresponding to the intra-layer autocorrelation. The partial derivatives are calculated as follows: Classical convolution operations can obtain features at different levels of abstraction and scales. In the feature learning module, single-layer autocorrelation information and multi-layer statistics of CNNs are used to obtain cross-correlation features between different layers, such as... Figure 3 As shown, two correlation operations were designed to obtain higher-order semantic features, including intra-layer autocorrelation features and inter-layer cross-correlation features. During model training, the feature matrix corresponding to the inter-layer cross-correlation features... and The partial derivatives are calculated as follows: By fusing intra-layer autocorrelation and inter-layer cross-correlation features, a deep feature vector containing high-order statistical information is generated, which serves as the basic feature for hash encoding.

[0028] S104, using the higher-order feature vector as input, constructing a multi-similarity loss by combining the semantic similarity matrix, and forming an overall cost function with the binary quantization loss, to train a deep hash model and learn hash encoding.

[0029] Self-similarity is the most widely used similarity metric based on sample pairs in learning. For hash learning, a higher self-similarity between negative sample pairs indicates a greater likelihood of unlearned sample information, which can mitigate the negative impact of imbalanced sample pairs on the hash model. Positive relative similarity represents the similarity between a positive sample and the anchor sample, while negative relative similarity measures the similarity between other negative samples and the anchor sample. For a study of these three types of similarity, see [link to study]. Figure 5 In this embodiment, a multi-similarity loss function is used for deep supervised hashing. This algorithm considers all similarities and achieves efficient sample training.

[0030] For a given image pair and and the corresponding binary encoding and Hamming distance inner product There is a relationship between them: The similarity of image pairs is quantified using the inner product: .

[0031] The defined semantic similarity matrix is ​​generated by calculating label-level and semantic-level similarity, and it can represent the differences in fine-grained supervision information other than label-level information. Based on similarity quantification using the inner product, multiplying it by the semantic similarity yields the optimal similarity between image pairs: .

[0032] By comparing samples with the negative samples that have the highest similarity, sample pairs are selected from the positive sample dataset, and the positive relative similarity is measured. Assumption For an anchor sample, The set of selected positive sample pairs. If If the following constraints are met, then positive sample pairs The positive sample will be selected, while the rest will be discarded.

[0033] It is a given threshold, and in this embodiment it will be Setting it to 0.5, similarly, if the similarity between image pairs is greater than the minimum similarity between positive sample pairs, then a negative sample pair should be selected. .

[0034] in: To maximize the calculated result for all semantically similar images, This indicates that the minimum value is taken for all semantically dissimilar images.

[0035] Simultaneously, self-similarity is learned by directly calculating pairwise similarity: Because negative relative similarity takes into account negative sample pairs The relative similarity between the negative sample pairs and the other remaining negative sample pairs is required, therefore another negative sample pair needs to be considered. The comparisons are made, and the relative similarity is defined as follows.

[0036] Furthermore, for negative sample pairs We introduced thresholds into the self-similarity and weight metrics. and The weights can be calculated. .

[0037] Weights are obtained by calculating self-similarity and negative relative similarity. Therefore, the parameters are... As The scaling factor. For positive sample pairs. weights The calculation is as follows.

[0038] By introducing semantic similarity into the multi-similarity loss function, the partial derivative of the multi-similarity loss function is the weight obtained above. The multi-similarity loss of deep supervised hashing can be expressed as: in: For For the set of all negative sample pairs of the anchor sample, For For the set of all positive sample pairs of the anchor sample, The scaling factor hyperparameter is used to calculate the weights of positive sample pairs. and The threshold hyperparameter is calculated for the weights of negative sample pairs.

[0039] The aforementioned multi-similarity loss can also be expressed in the following form: Derivation of partial derivatives: make but Similarity Find the partial derivatives because Derivation of partial derivatives: make but Similarity Find the partial derivatives because In this embodiment, a regularization term is added to the hash stream to ensure that the network output is close to the standard binary encoding.

[0040] Where: 1 represents the identity matrix, Representing vectors L 1-norm, It is the absolute value of the element. and This is the feature vector output by the hash stream. The overall cost function, calculated jointly by the multi-similarity loss and quantization error loss, is as follows: Here, λ is a weighting coefficient used to control the quantization error.

[0041] S105, generate the hash code corresponding to the query image, generate a list of search results based on the hash code of the query image and the hash codes of the candidate images, and complete the image search.

[0042] After the deep hash model is optimized, when image retrieval is needed for the query image, each input original image is mapped to... q -bit hash encoding. In this process, a CNN network structure based on high-order statistical learning is used to encode the input image into -bit hash encoding. D The feature vectors are 3D. Then, an element-wise transformation function is used. Transform the feature vector into q -bit hash encoding, where Let represent a symbolic function. For a given query, a retrieval list is learned by sorting the Hamming distances between the query image and candidate images. The Hamming distance between the hash codes of the query image and the candidate images is calculated as follows: in: and Image pairs and binary encoding, Length of binary hash code To evaluate the deep hashing image retrieval method proposed in this embodiment, extensive experiments were conducted on three datasets: MS-COCO, NUS-WIDE, and MIRFLICKR-25K, verifying the superiority of the proposed method.

[0043] The MS-COCO dataset contains 82,783 training samples and 40,504 test samples, covering 80 categories. After removing some images without category information, 122,218 image samples were obtained by combining the training and test samples. The NUS-WIDE dataset is a multi-label dataset containing over 270,000 images with 81 class labels. Each image is labeled using one or more of these 81 labels. Furthermore, this experiment selected the most commonly used categories as experimental data, randomly selecting labels for each category, resulting in 158,078 samples. The MIRFLICKR-25K dataset contains over 25,000 multi-label images with 38 class labels, with each image containing an average of 4.7 class labels from the 38 categories. The labels cover food, trees, people, and semantic scenes, among others.

[0044] For the MS-COCO and NUS-WIDE datasets, we randomly selected 5000 images as the test set, and set the remaining data as the query database. We then randomly selected 10000 samples from the query database as the training dataset for the model. For the MIRFLICKR-25K dataset, similar to the experimental setup in DHN [], we randomly selected 1000 samples as the test set and 4000 samples as the training set. All image sizes were then adjusted to [missing information]. This ensures that the data meets the input size of the underlying network.

[0045] To validate the effectiveness of our proposed method, we conducted detailed comparisons with several state-of-the-art supervised hashing methods on three public datasets, including two deep hashing methods, Deep Hash Network (DHN) and HashNet, and two non-deep hashing methods, Spectral Hash (SH) and Iterative Quantization (ITQ). For fairness, features obtained from CNNs were used as input to traditional hashing.

[0046] In the experiments, several evaluation metrics widely used in image hashing applications were employed to assess the effectiveness of our method, including Normalized Decay Cumulative Gain (NDCG), Average Cumulative Gain (ACG), Weighted Mean Average Precision (weighted mAP), accuracy within a Hamming radius of 2 (P@H≤2), Top N Precision (TopN-Precision), and Precision-Recall. Similar to previous multi-label hashing methods, Jaccard coefficients were used (…). To measure image pairs and Multi-level semantic similarity between them.

[0047] The method proposed in this embodiment uses a pre-trained AlexNet as the backbone network to obtain fine-grained features. A high-order statistical learning layer representing global semantic information is established after convolutional operations, and a hash layer is used instead of the final fully connected layer. During training, we set the mini-batch stochastic gradient descent momentum term to 0.9, the batch size to 256, and the weight decay to 0.0005. The learning rate is initialized to 0.0003 and reduced by 50% every 5000 iterations. Furthermore, based on experience, the hyperparameters α, β, γ, and λ are set to 0.9, 0.5, 0.4, and 0.1, respectively.

[0048] To conduct a comprehensive evaluation and comparison, the NDCG@1000, ACG@1000, and weightedmAP values ​​were first calculated on three publicly available benchmark datasets for different hash encoding lengths. Furthermore, three performance metrics for different hashing methods were evaluated: the P@H≤2 curve, the TopN-precision curve, and the precision-recall curve.

[0049] Tables 1 to 3 list the NDCG@1000, ACG@1000, and weightedmAP scores of the hash model on the MS-COCO, NUS-WIDE, and MIRFLICKR-25K datasets, respectively, demonstrating that the method proposed in this embodiment outperforms other techniques. Comparative experimental results show that deep hashing achieves significant performance improvements over traditional hashing algorithms at different hash code lengths. This is because deep hashing combines feature extraction and binary code learning within a shared end-to-end model, which facilitates the generation of binary codes with high-order similarity. For the deep hashing method, the proposed method in this embodiment integrates label-level and semantic-level similarity learning to guide the hash code learning process, resulting in an average performance improvement of 0.3%–2.3% over other hashing methods at different hash code lengths. Furthermore, compared to other hashing methods, this method extracts auto-related information from intra-layer features and cross-related information from inter-layer features, achieving stable and robust retrieval performance.

[0050] Table 1 Comparison Results of MS-COCO Datasets Table 2 Comparison results on the NUS-WIDE dataset Table 3 Comparison results on the MIRFLICKR-25K dataset We computed P@H≤2, TopN precision, and precision-recall curves based on 48-bit hash encoding on the NUS-WIDE dataset. Figures 6-8 As shown, by utilizing high-order statistical learning and fine-grained semantic similarity discriminative representation, our proposed hash model can generate hash codes for image retrieval discrimination, resulting in significant improvements in P@H≤2, TopN accuracy, and accuracy-recall rate in this embodiment.

[0051] parameter α and β These are two hyperparameters in our multi-layer similarity loss model that balance different similarity weights. Different parameters will affect the convergence of the hash model. To explore the impact of these parameters on model performance, we set them while keeping other hyperparameters fixed. = The weighted mAP scores were calculated using the formulas {0.1, 0.3, 0.5, 0.7, 0.9} and on the MS-COCO, NUS-WIDE, and MIRFLICKR-25K datasets, respectively. Experimental results are shown below. Figures 9-11 As shown, through analysis, when the parameter and As the size increases, the overall performance of our method remains stable. By setting... =0.9 and =0.5, the proposed model in this embodiment achieves the best overall performance.

[0052] It is the threshold for multi-similarity loss. This is achieved by setting... The parameter sensitivity experiments were conducted on the MS-COCO, NUS-WIDE, and MIRFLICKR-25K datasets with the expression ={0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0}.

[0053] like Figures 12-14 As shown, when the parameter When the threshold is greater than 0.4, increasing the semantic similarity boundary threshold leads to redundant information in different bits within the generated hash code, resulting in a significant performance degradation. According to the analysis results, when... When set to 0.4, the model in our proposed embodiment converges to the global optimum. Essentially, the quantization error loss is a regularizer that maps real-valued outputs to desired discrete values.

[0054] like Figures 15-17 As shown, this demonstrates the performance of 64-bit hash encoding on three benchmark datasets when... The weighted mAP is calculated for the values ​​of {0.3, 0.4, 0.5, 0.6, 0.7, 0.1, 0.2, 0.8, 0.9, 1.0}. Experimental results show that... When set to 1.0, the retrieval performance of this embodiment is optimal.

[0055] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0056] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A semantic fusion-based deep hashing image retrieval method, characterized in that, include: Obtain an image dataset with class labels and initialize a depth hashing model, then adjust the input image size according to the initial depth hashing model; The combined label-level and semantic-level similarity of images generates a semantic similarity matrix representing fine-grained differences between the images; Convolutional neural networks based on correlation operations extract intra-layer autocorrelation features and inter-layer cross-correlation features from the depth features of images, generating high-order feature vectors containing high-order statistical information. Using the higher-order feature vector as input, a multi-similarity loss is constructed by combining the semantic similarity matrix, and the overall cost function is formed by combining it with the binary quantization loss. This is used to train a deep hash model and learn hash encoding. Generate the hash code corresponding to the query image, and generate a list of search results based on the hash codes of the query image and the candidate images to complete the image retrieval.

2. The semantic fusion-based deep hashing image retrieval method according to claim 1, characterized in that, The steps of obtaining an image dataset with class labels, initializing a depth hashing model, and adjusting the input image size based on the initial depth hashing model include: Determine a training set containing N original images. and a corresponding tag set containing C categories ; The deep hashing model was initialized using AlexNet pre-trained on ImageNet as the backbone CNN network; Adjust all images to the input size suitable for the backbone CNN network, and complete the partitioning of the dataset into training, testing, and query databases.

3. The semantic fusion-based deep hashing image retrieval method according to claim 2, characterized in that, The label-level similarity and semantic-level similarity of the joint images generate a semantic similarity matrix representing fine-grained differences between the images, including: Extract each pair of images from the training set. and tag set and and determine the and semantic feature vector and ; by and Intersection and Union As a label-level similarity, and with and cosine similarity As a semantic-level similarity; The semantic similarity of image pairs is obtained by fusing the label-level similarity and semantic-level similarity. : in: As the core quantitative indicator of fusion-type similarity, These are the weighting coefficients. express and The inner product, The norm of a vector; Iterate through all image pairs to construct a semantic similarity matrix. .

4. The semantic fusion-based deep hashing image retrieval method according to claim 2, characterized in that, The convolutional neural network based on correlation operations extracts intra-layer autocorrelation features and inter-layer cross-correlation features from the depth features of the image, generating a high-order feature vector containing high-order statistical information, including: The preprocessed image is input into two independent CNN models, and the feature matrix is ​​obtained through convolution operations. and ; The and The input is fed into the bilinear pooling module to generate inter-layer cross-correlation features. Intra-layer autocorrelation features ; Fusion of the and Generate high-order feature vectors containing high-order statistical information.

5. The semantic fusion-based deep hashing image retrieval method according to claim 4, characterized in that, For higher-order statistical learning, the parameters are optimized using the backpropagation algorithm, and the feature matrix corresponding to the intra-layer autocorrelation is obtained. The partial derivatives are calculated as follows: Feature matrix corresponding to inter-layer cross-correlation features and The partial derivatives are calculated as follows: in: This is the global cost function.

6. The semantic fusion-based deep hashing image retrieval method according to claim 4, characterized in that, The process of using the higher-order feature vector as input, constructing a multi-similarity loss by combining the semantic similarity matrix, and combining this with the overall cost function formed by binary quantization loss to train a deep hash model and learn hash encoding includes: Combine semantic similarity matrix to filter valid positive and negative sample pairs; Define self-similarity and negative relative similarity, calculate the weights for positive and negative sample pairs respectively, and implement differentiated weighting for different sample pairs; In multiple similarity loss functions Semantic similarity is introduced to optimize the similarity of sample pairs. The partial derivatives are used to calculate the weights for positive and negative sample pairs; Add regularized binary quantization loss The L1 norm-constrained hash encoding output is close to standard binary form: Where: 1 represents the identity matrix, Representing vectors L 1-norm, It is the absolute value of the element. and The feature vector output by the hash stream; By introducing weighting coefficients to balance the effects of multi-similarity loss and quantization loss, the overall cost function is obtained through fusion: Where: λ is the weighting coefficient that controls the quantization error.

7. The semantic fusion-based deep hashing image retrieval method according to claim 6, characterized in that, The combination of semantic similarity matrix Filtering valid positive and negative sample pairs includes: Based on the relationship between Hamming distance and inner product, the inner product is used to define the similarity of image pairs: ,in: and Image pairs and binary encoding; Combined with semantic similarity To obtain the optimal similarity between image pairs: ; Set threshold Filter positive and negative sample pairs. Positive sample pairs must meet the following conditions: Negative sample pairs must meet the following conditions: in: To maximize the calculated result for all semantically similar images, This indicates that the minimum value is taken for all semantically dissimilar images.

8. The semantic fusion-based deep hashing image retrieval method according to claim 7, characterized in that, The definition of self-similarity and negative relative similarity is used to calculate the weights for positive and negative sample pairs, respectively, to achieve differentiated weighting for different sample pairs, including: By directly calculating pairwise similarity and negative sample pairs Difference defines self-similarity and negative relative similarity ; For negative sample pairs The weights for negative sample pairs are calculated as follows: For positive sample pairs The weights for positive sample pairs are calculated as follows: in: For For the set of all negative sample pairs of the anchor sample, For For the set of all positive sample pairs of the anchor sample, The scaling factor hyperparameter is used to calculate the weights of positive sample pairs. and The threshold hyperparameter is calculated for the weights of negative sample pairs.

9. The semantic fusion-based deep hashing image retrieval method according to claim 8, characterized in that, The for: 。 10. The semantic fusion-based deep hashing image retrieval method according to claim 1, characterized in that, The process of generating the hash code corresponding to the query image, generating a search result list based on the query image hash code and the candidate image hash codes, and completing the image retrieval includes: The deep hash model is used to generate the binary hash code corresponding to the query image; Calculate the Hamming distance between the query image hash code and the candidate image hash codes: in: and Image pairs and binary encoding, The length of the binary hash code; Candidate images are sorted by Hamming distance from smallest to largest to generate a list of search results, thus completing the image retrieval.

Images