Image retrieval method and device based on depth difficulty perceptual hash and storage medium

Abstract

The invention relates to the technical field of information retrieval, in particular to an image retrieval method and device based on depth difficulty perceptual hash and a storage medium. In the method, in order to comprehensively model a global geometric structure of an embedded space, self-adaptive difficulty perception enhancement is carried out on an original data sample, and the difficulty level is dynamically adjusted by utilizing linear interpolation for embedding; difficulty perception feature synthesis is carried out to generate enhanced data samples which are consistent in semantics and reserved in labels, and loop training is carried out on the enhanced data samples; the strategy not only makes full use of potential information in all training samples, but also systematically challenges learned metrics with the difficulty of adaptive calibration, ensures continuous optimization on the whole feature manifold, and promotes generation of similarity-preserving hash codes.

Description

Technical Field

[0001] The present invention relates to the technical field of information retrieval, and in particular to an image retrieval method, device, and storage medium based on deep hard perception hashing. Background Art

[0002] With the rapid development of multimedia technology and intelligent devices, the way people obtain multimedia data has become more convenient and diversified. A large number of multimedia data mainly including images, texts, and videos are continuously uploaded to the Internet. How to efficiently and accurately retrieve the information of interest to users from the huge multimedia data set has become a hot research issue.

[0003] Due to advantages such as high efficiency, good scalability, and resource savings, the approximate nearest neighbor (ANN) search algorithm has been widely used in large-scale data retrieval. Among them, hashing learning, as one of the classic methods in ANN, has received extensive attention due to its excellent retrieval speed, accuracy, and low storage overhead. Hashing learning maps high-dimensional image features to a low-dimensional Hamming space, effectively reducing the computational complexity while maintaining the similarity between images. Its core idea is to use efficient exclusive-or operations to measure the similarity between samples, thus greatly improving the retrieval efficiency and reducing the computational cost. With the vigorous development of deep learning technology, neural networks have gradually been introduced into the field of hashing learning. Researchers use deep neural networks for image feature extraction and integrate them into a unified framework with the hashing coding process, thus forming a deep hashing learning method. This method can generate hashing codes with compact intra-class features and separated inter-class features, showing significant advantages in improving retrieval performance.

[0004] Although the existing deep cross-modal hashing algorithms have made significant progress, there are still some limitations. When training a deep hashing model, most of the existing deep hashing learning methods adopt a hard negative sample mining strategy to solve the problem of insufficient training samples. These strategies usually undersample the training set to extract challenging and informative samples, thus generating a large number of gradients. However, the hard negative sample mining strategy only selects from a subset of samples, which may cause some data points to be resampled repeatedly, while other data points may never be sampled. This may lead to overfitting of the metric space near the oversampled data points and underfitting near the undersampled data points. Summary of the Invention

[0005] To solve at least one problem in the prior art, the present invention provides an image retrieval method, device, and storage medium based on deep hard perception hashing.

[0006] The first aspect of the present invention provides a method for training an image retrieval model based on deep hard-aware hashing, which includes the following steps: Obtain original data samples, where the original data samples include anchor samples, positive samples, positive sample labels, negative samples, and negative sample labels; Map the original data samples from the feature space to the metric space (or embedding space); In the metric space, generate hard negative samples based on the anchor samples and negative samples, where the distance between the hard negative samples and the anchor samples is less than the distance between the negative samples and the anchor samples; Based on the anchor samples, positive samples, positive sample labels, hard negative samples, and negative sample labels in the metric space, generate enhanced data samples in the feature space through a generator, where the enhanced data samples include anchor samples, positive samples, positive sample labels, enhanced negative samples, and negative sample labels; Train a convolutional neural network with the enhanced data samples, update the parameters of the convolutional neural network according to the objective loss, and obtain an image retrieval model.

[0007] In this method, hard negative samples are obtained according to the distance between samples in the metric space, and then mapped back to the feature space. The obtained enhanced data samples retain the difficulty information of the hard negative samples and also retain the original label information. Such enhanced data samples provide valuable information for the iterative training of the convolutional neural network, so that an image retrieval model with higher accuracy can be obtained. In the metric space, the distance between a sample (hard negative sample or negative sample) and an anchor sample is calculated to control the difficulty level of the hard negative sample. For example, reducing the distance between the hard negative sample and the anchor sample can increase the difficulty of the hard negative sample. The strategy of generating hard negative samples avoids the situation where some data points are repeatedly sampled and other data points may never be sampled, and solves the problems of overfitting near oversampled data points and underfitting near undersampled data points.

[0008] In some alternative embodiments, the hard negative sample is calculated by the following formula: ; where, represents the anchor sample, represents the negative sample, represents the hard negative sample; represents the Euclidean distance between the anchor sample and the negative sample; represents a hyperparameter; represents the average training loss in the most recent training epoch, represents the scaling factor for adjusting the influence strength.

[0009] In some optional embodiments, the parameters of the generator are updated according to the generation loss, where the generation loss is calculated as follows: ; in, represents the generation loss, represents the reconstruction loss, represents the softmax loss, represents the balance factor; Represents the original data sample features in the metric space; Represents the synthetic features of the original data sample features after the generator; Represents the enhanced data sample features; Indicates the target category.

[0010] In some optional embodiments, the softmax loss is calculated as follows: ; in, Represents the enhanced data sample features; ,express The corresponding original data sample label; Indicates the total number of categories; Indicates that Mapped to the output value of the target category; Indicates that Mapped to the sum of the output values of all categories; represents the natural exponential function.

[0011] In some optional embodiments, the target loss is calculated as follows: ; in, Indicates target loss; Represents a preset parameter. represents the triplet loss calculated on the original data sample, represents the triplet loss calculated on the augmented data samples.

[0012] In some optional embodiments, Calculated by the following formula: ; in, represents the Euclidean distance, Indicates the set interval boundary; represents the anchor point sample, represents a positive sample, Represents negative samples.

[0013] In some alternative embodiments, is calculated by the following formula: ; where represents the Euclidean distance, represents the set interval boundary; represents the anchor sample, represents the positive sample, represents the hard negative sample.

[0014] In some alternative embodiments, , .

[0015] The second aspect of the present invention provides an image retrieval method based on deep hard-aware hashing, which includes the following steps: Input the image to be retrieved into the image retrieval model obtained by the above-mentioned image retrieval model training method based on deep hard-aware hashing, and the image retrieval model outputs the retrieval result.

[0016] The third aspect of the present invention provides an electronic device, which includes a memory and a processor. Among them, the memory is used to store computer programs; the processor is used to execute the above-mentioned image retrieval model training method based on deep hard-aware hashing or the image retrieval method based on deep hard-aware hashing according to the computer programs.

[0017] The third aspect of the present invention provides a computer-readable storage medium. Among them, the computer-readable storage medium is used to store computer programs; the computer programs, when executed by the processor, implement the above-mentioned image retrieval model training method based on deep hard-aware hashing or the image retrieval method based on deep hard-aware hashing.

[0018] The technical solution of the embodiment of the present invention has the following beneficial effects: In order to comprehensively model the global geometric structure of the embedding space, adaptive hard-aware enhancement is performed on the original data samples, and the difficulty level is dynamically adjusted by using linear interpolation of the embedding; and hard-aware feature synthesis is performed to generate enhanced data samples with consistent semantics and label retention for cyclic training; this strategy not only fully utilizes the potential information in all training samples, but also systematically challenges the learned metric with an adaptively calibrated difficulty, ensuring continuous optimization on the entire feature manifold and promoting the generation of similarity-preserving hash codes. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the network structure of DHAH in the embodiment of the present invention.

[0020] Figure 2Schematic diagram of the adaptive hard perception enhancement module in DHAH in the embodiments of the present invention.

[0021] Figure 3 Schematic diagram of the triplet loss in DHAH in the embodiments of the present invention.

[0022] Figure 4 Precision-recall curve and TopN precision curve of different models on the MIRFLICKR-25K dataset; among them, (a) is the precision-recall curve, and (b) is the TopN precision curve.

[0023] Figure 5 Precision-recall curve and TopN precision curve of different models on the NUS-WIDE dataset; among them, (a) is the precision-recall curve, and (b) is the TopN precision curve.

[0024] Figure 6 Precision-recall curve and TopN precision curve of different models on the MS COCO dataset; among them, (a) is the precision-recall curve, and (b) is the TopN precision curve. Detailed implementation manners

[0025] Next, the technical solutions in the embodiments of the present invention will be described clearly and completely in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0026] According to the embodiments of the present invention, first, an image retrieval model is trained by a training method for an image retrieval model based on deep hard perception hashing, and then the image retrieval model is used for image retrieval. For the convenience of representation, the image retrieval model is denoted as DHAH.

[0027] As Figure 1 shown, DHAH includes a convolutional neural network (CNN), an adaptive hard perception enhancement module, and a hard perception feature synthesis module, where the CNN is used to extract features from images; the adaptive hard perception enhancement module adjusts negative samples to be more difficult through linear interpolation to obtain high-difficulty negative samples; the hard perception feature synthesis module is a generator, and the generator is used to generate more difficult enhanced negative samples.

[0028] During training, first obtain the original data samples, where the original data samples include anchor samples, positive samples, positive sample labels, negative samples, and negative sample labels. These samples can all be images, and the labels are used to indicate the target categories. Positive samples and anchor samples belong to the same target category, while negative samples and anchor samples belong to different target categories.

[0029] Secondly, map the original data samples from the feature space to the metric space (or embedding space). This operation can be achieved through a mapping function to project the original data samples from the feature space onto a metric space .

[0030] As Figure 2 shown, in the metric space, based on the anchor sample and the negative sample, generate hard negative samples, where the distance between the hard negative sample and the anchor sample is less than the distance between the negative sample and the anchor sample. In this metric space, adjust and construct difficulty-aware tuples according to the distance between samples. The change in distance will directly affect the difficulty level of the tuples. For example, reducing the distance between negative sample pairs can increase their difficulty.

[0031] In a sample set, usually more negative sample pairs can be constructed than positive sample pairs. Therefore, for simplicity of processing, only focus on adjusting the distance of negative sample pairs, and the samples in other tuples remain unchanged. Specifically, given a negative sample pair (the anchor sample and its corresponding negative sample ), generate an enhanced and more challenging hard negative sample through linear interpolation. Subsequently, use the average training loss to regulate the difficulty level, and this average training loss can effectively reflect the training process. When the average training loss is small, it means that the enhanced negative sample is closer to the anchor sample, thus forming more challenging tuples during the training process. This process can be expressed as: ; where, represents the anchor sample, represents the negative sample, represents the hard negative sample; , represents the Euclidean distance between the anchor sample and the negative sample; represents a hyperparameter used to set the reference distance for the operation scale, such as the distance between positive sample pairs or a preset value; represents the average training loss in the most recent training epoch, represents the scaling factor used to adjust the influence intensity.

[0032] After obtaining the difficulty-aware tuples in the metric space, the next goal is to map them back to the feature space for model training. Based on the anchor samples, positive samples, positive sample labels, high-difficulty negative samples, and negative sample labels in the metric space, the generator generates augmented data samples in the feature space, where the augmented data samples include anchor samples, positive samples, positive sample labels, augmented negative samples, and negative sample labels. Specifically, a generator is trained that can map the augmented embedded features in the tuple back to the original feature space for subsequent training processes. Since it is usually difficult for the generator to perfectly map all embedded features in practical applications, in order to ensure the usability and consistency of the synthetic features in the feature space, not only the augmented negative samples are mapped, but also the rest of the unchanged samples in the tuple are mapped together, so as to maintain the overall consistency and trainability of the tuple, as shown in the following formula: ; where, and represent the tuples in the feature space and the embedded space respectively; represents the generator; represents the parameters of the generator.

[0033] In order to effectively utilize the difficulty-aware synthetic features in training , it is necessary to ensure that they retain both the class labels of the original samples and the augmented "difficulty" characteristics. This is constrained by the generation loss function, which consists of two parts: the reconstruction loss function and the softmax loss function. In order to make the synthetic high-difficulty negative samples semantically consistent with the augmented high-difficulty negative samples, it must form a valid tuple that meets the specified "difficulty level". Therefore, the optimization objective of the generator is as follows: ; where, represents the generation loss, represents the reconstruction loss, represents the softmax loss, represents the balance factor; represents the feature of the original data samples in the metric space (the set of feature vectors extracted from the image by the convolutional neural network), represents the synthetic feature of the original data sample feature (un-augmented embedding) passing through the generator (the result of the generator mapping the original embedding back to the feature space), Represents the enhanced data sample features (the result of mapping the enhanced embedding back to the feature space by the generator, i.e., the difficulty-aware synthetic features); Represents the target class. The softmax loss is calculated as follows: ; ; Where, Represents the enhanced data sample features; , represents The corresponding original data sample label (target class); Represents the total number of classes; Represents mapping To the output value of the target class; Represents mapping To the sum of the output values of all classes; Represents the natural exponential function.

[0034] Reconstruction loss Is used to force the generator to learn a "lossless" mapping relationship by minimizing the difference between the original features and the reconstructed features, ensuring that the hardness adjustment (such as distance change) in the embedding space can be accurately reflected in the feature space. Making each sample point as close as possible to its original position during the mapping process; at the same time, the softmax loss Ensures that the synthesized enhanced features can retain the original semantic labels.

[0035] After obtaining the enhanced data samples that retain the original labels, they are further used to train DHAH. Train a convolutional neural network with the enhanced data samples, and update the parameters of the convolutional neural network according to the target loss to obtain DHAH. Where, the target loss function of DHAH is shown as follows: ; Where, Represents the target loss; Represents a preset parameter, Represents the triplet loss calculated for the original data samples, Represents the triplet loss calculated for the enhanced data samples. In this embodiment, .

[0036] Introduce As a balancing factor, used to appropriately reduce the weight allocation to the synthetic features when the generator has not been fully trained, that is, when the generation loss Is relatively high, because the synthetic features generated at this time may lack sufficient authenticity and semantic relevance. And Are both triplet loss functions, respectively defined as: ; ; Among them, represents the Euclidean distance, represents the set margin. In this embodiment, . Figure 3 shows a schematic of the triplet loss.

[0037] It should be noted that the above process generates synthetic features that retain both the "difficulty" information and semantic labels of the samples. The enhanced features are mapped back from the embedding space to the feature space using a simultaneously trained generator, while retaining the labels and enhanced difficulty information to supplement the original embedding features, thereby making full use of the information hidden in all samples. DHAH does not directly generate discriminative hash codes that retain semantic information after optimizing the CNN, but only obtains continuous class hash codes with different bits within [-1, 1]. In addition, the test samples are input into the trained network, and the element-wise transformation symbol sign() is used to project the continuous real-valued features into the corresponding discrete codes.

[0038] Finally, the image to be retrieved is input into the trained DHAH to achieve image retrieval.

[0039] To demonstrate the performance of the image retrieval model (DHAH) in the embodiments of the present invention, experiments are verified using the MIRFLICKR-25K dataset, NUS-WIDE dataset, and MS COCO dataset. The experiments use the open-source PyTorch library and are conducted on a Linux server equipped with an NVIDIA RTX 3090 GPU.

[0040] Ten typical deep cross-modal image retrieval models are selected for comparison with the image retrieval model (abbreviated as DHAH) in the embodiments of the present invention. They are Deep Supervised Hashing (DSH), HashNet, OrthoHash, Improved Deep Hashing (IDHN), Hybrid Proxy Pair Loss (HyP 2 Loss), Hashing Hinge Function (HHF), CenterHash, Causal Feature Learning to Binary Injection Hashing Learning (CFBH), Deep Consistent Penalty Hashing with Noisy Robust Representations (DCPH), Distribution-Consistency Guided Multimodal Hashing (DCGMH). The models for comparison are based on the official source code, and the relevant parameter settings refer to the original papers.

[0041] To evaluate the performance of the model in hashing retrieval, three commonly used evaluation metrics are employed: mean average precision (mAP), precision-recall (PR) curve, and TopN precision curve. The mAP score represents the average accuracy of the model across the entire dataset and is a widely recognized standard performance metric in the field of information retrieval. The PR curve demonstrates the relationship between recall and precision and can provide insights into the overall retrieval performance. The TopN precision curve measures the accuracy based on the returned Top-N samples.

[0042] The mean average precision (mAP) performance of different models on the MIRFLICKR-25K, NUS-WIDE, and MS COCO datasets is shown in Tables 1 - 3. It can be seen that DHAH significantly outperforms other models.

[0043] Table 1 mAP of different models on the MIRFLICKR-25K dataset for 16-bit, 32-bit, 64-bit, and 128-bit

[0044] In Table 1, MIRFLICKR-25K@all means sorting with all the data in the MIRFLICKR-25K library.

[0045] Table 2 mAP of different models on the NUS-WIDE dataset for 16-bit, 32-bit, 64-bit, and 128-bit

[0046] In Table 2, NUS-WIDE@5000 means sorting with 5000 data in the NUS-WIDE library.

[0047] Table 3 mAP of different models on the MS COCO dataset for 16-bit, 32-bit, 64-bit, and 128-bit

[0048] In Table 3, MS COCO@all means sorting with all the data in the MS COCO library.

[0049] In addition to the mAP comparison, the effectiveness of DHAH is further evaluated by plotting the 32-bit precision-recall (PR) curve and the TopN accuracy curve, as Figures 4 - 6 shown.

[0050] To evaluate the effectiveness of each module in DHAH, ablation experiments were conducted for verification. Three variants were designed: DPSH-pairwise, DSH-triplewise, and DHAH-pairwise, and a simplified study was carried out on the MIRFLICKR-25K dataset. Each variant modified the original DHAH framework. Specifically, DPSH-pairwise replaced the objective loss function in DHAH with the pairwise loss function used in DPSH (Li W J, Wang S, Kang W C. Feature learning based deep supervised hashing with pairwise labels[J]. arXiv preprint arXiv:1511.03855,2015.); DSH-triplewise replaced the objective loss function in DHAH with the triplet loss function in DSH (Liu H, Wang R, Shan S, et al. Deep supervised hashing for fast image retrieval[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 2064-2072.); DHAH-pairwise replaced the triplet-based loss in DHAH with a pairwise loss function. Table 4 shows the comparison of the mean average precision (mAP) results of these variants with the original DHAH. It can be seen that the performance of each variant is lower than that of the complete DHAH architecture. It is worth noting that DHAH performs better than DSH-triplewise, and DHAH-pairwise performs better than DPSH-pairwise, which proves the effectiveness of the objective loss function in DHAH.

[0051] Table 4 mAP of DHAH and its variants at 16-bit, 32-bit, 64-bit, and 128-bit on the MIRFLICKR-25K dataset

[0052] In summary, DHAH has excellent image retrieval performance. This excellent performance is attributed to the introduction of the adaptive hard perception enhancement module, which strengthens the representation learning. Moreover, the hard perception feature synthesis module generates synthetic data that preserves labels for repeated training. In addition, the objective loss function promotes model optimization. This enables the model to fully utilize the potential information contained in all samples, thus achieving the best performance.

[0053] The above are only the preferred embodiments of the present invention, and do not impose any formal or substantial limitations on the present invention. It should be noted that for those of ordinary skill in the art of this technology, without departing from the method of the present invention, several improvements and supplements can still be made, and these improvements and supplements should also be regarded as the protection scope of the present invention. All those who are familiar with the technology in this field, without departing from the spirit and scope of the present invention, when making some minor changes, modifications and equivalent variations of evolution using the technical content disclosed above, are all equivalent embodiments of the present invention; at the same time, any equivalent changes, modifications and evolutions made to the above embodiments based on the substantial technology of the present invention still fall within the scope of the technical solutions of the present invention.

Claims

1. A method for training an image retrieval model based on deep difficulty-aware hashing, characterized in that Including the following steps: Obtain original data samples, where the original data samples include anchor samples, positive samples, positive sample labels, negative samples, and negative sample labels; Map the original data samples from the feature space to the metric space; In the metric space, based on the anchor samples and negative samples, generate hard negative samples, where the distance between the hard negative samples and the anchor samples is less than the distance between the negative samples and the anchor samples; Based on the anchor samples, positive samples, positive sample labels, hard negative samples, and negative sample labels in the metric space, generate enhanced data samples in the feature space through a generator, where the enhanced data samples include anchor samples, positive samples, positive sample labels, enhanced negative samples, and negative sample labels; Train a convolutional neural network with the enhanced data samples, and update the parameters of the convolutional neural network according to the objective loss to obtain an image retrieval model.

2. The method according to claim 1, wherein The hard negative samples are calculated by the following formula: ； Among them, represents an anchor sample, represents a negative sample, represents a hard negative sample; , represents the Euclidean distance between the anchor sample and the negative sample; represents a hyperparameter; represents the average training loss in the most recent training epoch, represents for adjusting the scaling factor of the influence intensity.

3. The method according to claim 1, wherein Update the parameters of the generator according to the generation loss, where the generation loss is calculated by the following formula: ； Among them, represents the generation loss, represents the reconstruction loss, represents the softmax loss, represents the balance factor; represents the feature of the original data sample in the metric space; represents the synthetic feature of the original data sample feature through the generator; represents the feature of the augmented data sample; represents the target category.

4. The method according to claim 3, wherein The softmax loss is calculated by the following formula: ； Among them, represents enhanced data sample features; , represents the corresponding original data sample label; represents the total number of categories; represents that is mapped to the output value of the target category; represents that is mapped to the sum of the output values of all categories; represents the natural exponential function.

5. The method according to claim 1, characterized in that, The objective loss is calculated by the following formula: ； Wherein, represents the target loss; represents a preset parameter, represents the triplet loss calculated for the original data samples, represents the triplet loss calculated for the augmented data samples.

6. The method according to claim 1, wherein Calculated by the following formula: ； Among them, represents the Euclidean distance, represents the set interval boundary; represents the anchor point sample, represents the positive sample, represents the negative sample.

7. The method according to claim 1, characterized in that, Calculated by the following formula: ； Among them, represents the Euclidean distance, represents the set interval boundary; represents the anchor point sample, represents the positive sample, represents the high-difficulty negative sample.

8. An image retrieval method based on deep difficulty-aware hashing, characterized in that, Including the following steps: Input the image to be retrieved into the image retrieval model obtained by the method for training an image retrieval model based on deep hard-aware hashing according to any one of claims 1-7, and output the retrieval result by the image retrieval model.

9. An electronic device, characterized in that, Including a memory and a processor, wherein the memory is used to store a computer program; the processor is used to execute the method for training an image retrieval model based on deep hard-aware hashing according to any one of claims 1-7 or the method for image retrieval based on deep hard-aware hashing according to claim 8 according to the computer program.

10. A computer-readable storage medium, characterized in that, A computer-readable storage medium is used to store a computer program; the computer program, when executed by a processor, implements the method for training an image retrieval model based on deep hard-aware hashing according to any one of claims 1-7 or the method for image retrieval based on deep hard-aware hashing according to claim 8.

Images