Image retrieval method, device and storage medium based on deep difficulty-aware hashing

Abstract

The present invention relates to the field of information retrieval technology, and more particularly to an image retrieval method, device, and storage medium based on deep difficulty-aware hashing. In the present invention, adaptive difficulty-aware enhancement is performed on original data samples to comprehensively model the global geometric structure of the embedding space, dynamically adjusting the difficulty level by linear interpolation of the embedding. Difficulty-aware feature synthesis is also performed to generate semantically consistent and label-preserving enhanced data samples for cyclic training. This strategy not only fully utilizes the potential information in all training samples, but also systematically challenges the learned metric with an adaptively calibrated difficulty, ensuring continuous optimization across the entire feature manifold and promoting the generation of similarity-preserving hash codes.

Description

Technical Field

[0001] The present invention relates to the field of information retrieval technology, and in particular to an image retrieval method, device and storage medium based on deep difficulty-aware hashing. Background Art

[0002] With the rapid development of multimedia technology and smart devices, the ways people access multimedia data have become more convenient and diverse. Large amounts of multimedia data, primarily images, text, and videos, are continuously uploaded to the internet. How to efficiently and accurately retrieve information of interest from this massive multimedia dataset has become a hot research topic.

[0003] Approximate nearest neighbor (ANN) search algorithms have been widely used in large-scale data retrieval due to their high efficiency, good scalability, and resource-saving advantages. Among them, hash learning, a classic ANN method, has attracted widespread attention due to its excellent retrieval speed, accuracy, and low storage overhead. Hash learning maps high-dimensional image features into a low-dimensional hash space (Hamming space), effectively reducing computational complexity while maintaining inter-image similarity. Its core concept is to use the efficient XOR operation to measure the similarity between samples, significantly improving retrieval efficiency and reducing computational cost. With the rapid development of deep learning technology, neural networks have gradually been introduced into the field of hash learning. Researchers have applied deep neural networks to image feature extraction and integrated them with hash encoding processes into a unified framework, resulting in the deep hash learning method. This method can generate hash codes with compact intra-class features and separated inter-class features, showing significant advantages in improving retrieval performance.

[0004] Although existing deep cross-modal hashing algorithms have made significant progress, some limitations still exist. When training deep hashing models, most existing deep hashing learning methods use hard negative sample mining strategies to address the problem of insufficient training samples. These strategies typically undersample the training set to extract challenging and informative samples that produce a large number of gradients. However, hard negative sample mining strategies only select from a subset of samples, which may cause some data points to be repeatedly sampled, while other data points may never be sampled. This can cause the metric space to overfit near oversampled data points and underfit near undersampled data points. Summary of the Invention

[0005] In order to solve at least one problem in the prior art, the present invention provides an image retrieval method, device and storage medium based on deep difficulty-aware hashing.

[0006] A first aspect of the present invention provides an image retrieval model training method based on deep difficulty-aware hashing, which comprises the following steps:

[0007] Obtaining original data samples, where the original data samples include anchor samples, positive samples, positive sample labels, negative samples, and negative sample labels;

[0008] Mapping raw data samples from feature space to metric space (or embedding space);

[0009] In the metric space, based on the anchor samples and negative samples, high-difficulty negative samples are generated, where the distance between the high-difficulty negative samples and the anchor samples is smaller than the distance between the negative samples and the anchor samples;

[0010] Based on the anchor samples, positive samples, positive sample labels, high-difficulty negative samples and negative sample labels in the metric space, the generator generates enhanced data samples in the feature space, where the enhanced data samples include anchor samples, positive samples, positive sample labels, enhanced negative samples and negative sample labels;

[0011] The convolutional neural network is trained by augmenting data samples, and the parameters of the convolutional neural network are updated according to the target loss to obtain an image retrieval model.

[0012] In this method, high-difficulty negative samples are generated in metric space based on the distance between samples. These samples are then mapped back to the feature space. The resulting augmented data samples retain the difficulty information of the high-difficulty negative samples while also preserving the original label information. Such augmented data samples provide valuable information for the iterative training of convolutional neural networks, thereby achieving more accurate image retrieval models. In metric space, the distance between samples (high-difficulty negative samples or negative samples) and anchor samples is calculated to control the difficulty of high-difficulty negative samples. For example, reducing the distance between high-difficulty negative samples and anchor samples can increase the difficulty of high-difficulty negative samples. This strategy for generating high-difficulty negative samples avoids the situation where some data points are repeatedly sampled while others may never be sampled, thus solving the problem of overfitting near oversampled data points and underfitting near undersampled data points.

[0013] In some optional embodiments, the high-difficulty negative sample is calculated by the following formula:

[0014] ;

[0015] in, represents the anchor point sample, represents negative samples, Indicates high-difficulty negative samples; , represents the Euclidean distance between the anchor sample and the negative sample; represents a hyperparameter; represents the average training loss in the most recent training round, Indicates that it is used for adjustment Scaling factor affecting the intensity.

[0016] In some optional embodiments, the parameters of the generator are updated according to the generation loss, where the generation loss is calculated as follows:

[0017] ;

[0018] in, represents the generation loss, represents the reconstruction loss, represents the softmax loss, represents the balance factor; Represents the original data sample features in the metric space; Represents the synthetic features of the original data sample features after the generator; Represents the enhanced data sample features; Indicates the target category.

[0019] In some optional embodiments, the softmax loss is calculated as follows:

[0020] ;

[0021] in, Represents the enhanced data sample features; ,express The corresponding original data sample label; Indicates the total number of categories; Indicates that Mapped to the output value of the target category; Indicates that Mapped to the sum of the output values ​​of all categories; represents the natural exponential function.

[0022] In some optional embodiments, the target loss is calculated as follows:

[0023] ;

[0024] in, Indicates target loss; Represents a preset parameter. represents the triplet loss calculated on the original data sample, represents the triplet loss calculated on the augmented data samples.

[0025] In some optional embodiments, Calculated by the following formula:

[0026] ;

[0027] in, represents the Euclidean distance, Indicates the set interval boundary; represents the anchor point sample, represents a positive sample, Represents negative samples.

[0028] In some optional embodiments, Calculated by the following formula:

[0029] ;

[0030] in, represents the Euclidean distance, Indicates the set interval boundary; represents the anchor point sample, represents a positive sample, Represents high-difficulty negative samples.

[0031] In some optional embodiments, , .

[0032] A second aspect of the present invention provides an image retrieval method based on deep difficulty-aware hashing, which comprises the following steps:

[0033] The image to be retrieved is input into the image retrieval model obtained by the above-mentioned image retrieval model training method based on deep difficulty perceptual hashing, and the image retrieval model outputs the retrieval result.

[0034] A third aspect of the present invention provides an electronic device comprising a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the above-mentioned image retrieval model training method based on deep difficulty perceptual hashing or the image retrieval method based on deep difficulty perceptual hashing according to the computer program.

[0035] A third aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program; when the computer program is executed by a processor, it implements the above-mentioned image retrieval model training method based on deep difficulty perceptual hashing or the image retrieval method based on deep difficulty perceptual hashing.

[0036] The technical solution of the embodiments of the present invention has the following beneficial effects: in order to comprehensively model the global geometric structure of the embedding space, adaptive difficulty-aware enhancement is performed on the original data samples, and the difficulty level is dynamically adjusted by linear interpolation of the embedding; and difficulty-aware feature synthesis is performed to generate semantically consistent and label-preserving enhanced data samples for cyclic training; this strategy not only fully utilizes the potential information in all training samples, but also systematically challenges the learned metrics with adaptively calibrated difficulty, ensuring continuous optimization across the entire feature manifold and promoting the generation of similarity-preserving hash codes. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the network structure of DHAH in an embodiment of the present invention.

[0038] Figure 2 Schematic diagram of the adaptive difficulty perception enhancement module in DHAH in an embodiment of the present invention.

[0039] Figure 3 Schematic diagram of triplet loss in DHAH in an embodiment of the present invention.

[0040] Figure 4 The accuracy-recall curves and TopN accuracy curves of different models on the MIRFLICKR-25K dataset; (a) is the accuracy-recall curve, and (b) is the TopN accuracy curve.

[0041] Figure 5 The accuracy-recall curves and TopN accuracy curves of different models on the NUS-WIDE dataset; (a) is the accuracy-recall curve, and (b) is the TopN accuracy curve.

[0042] Figure 6 The accuracy-recall curves and TopN accuracy curves of different models on the MS COCO dataset; (a) is the accuracy-recall curve, and (b) is the TopN accuracy curve. DETAILED DESCRIPTION

[0043] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0044] According to an embodiment of the present invention, an image retrieval model is first trained using a deep difficulty-aware hashing (DHAH)-based image retrieval model training method, and then image retrieval is performed using the image retrieval model. For ease of representation, the image retrieval model is labeled DHAH.

[0045] like Figure 1 As shown in the figure, DHAH includes a convolutional neural network (CNN), an adaptive difficulty-aware enhancement module, and a difficulty-aware feature synthesis module, where CNN is used to extract features from images; the adaptive difficulty-aware enhancement module adjusts negative samples to be more difficult through linear interpolation to obtain high-difficulty negative samples; the difficulty-aware feature synthesis module is a generator that uses the generator to generate more difficult enhanced negative samples.

[0046] During training, we first obtain raw data samples, which include anchor samples, positive samples, positive sample labels, negative samples, and negative sample labels. These samples can all be images, and the labels are used to indicate the target category. Positive samples and anchor samples belong to the same target category, while negative samples and anchor samples belong to different target categories.

[0047] Secondly, the original data samples are mapped from the feature space to the metric space (or embedding space). This operation can be done through a mapping function To achieve this, the original data samples are transformed from the feature space Projection into a metric space middle.

[0048] like Figure 2 As shown in the figure, in the metric space, high-difficulty negative samples are generated based on anchor samples and negative samples, where the distance between the high-difficulty negative sample and the anchor sample is smaller than the distance between the negative sample and the anchor sample. In this metric space, difficulty-aware tuples are adjusted and constructed based on the distance between samples. Changes in distance directly affect the difficulty of the tuple. For example, reducing the distance between negative sample pairs can increase their difficulty.

[0049] In a sample set, more negative sample pairs can be constructed than positive sample pairs. Therefore, in order to simplify the process, we only focus on the distance adjustment of negative sample pairs, and the samples in other tuples remain unchanged. Specifically, given a negative sample pair (anchor sample and its corresponding negative sample ), generate an enhanced, more challenging high-difficulty negative sample through linear interpolation Subsequently, the difficulty level is regulated by using the average training loss, which effectively reflects the training process. When the average training loss is small, it means that the enhanced negative samples are closer to the anchor samples, thus forming more challenging tuples during the training process. This process can be expressed as:

[0050] ;

[0051] in, represents the anchor point sample, represents negative samples, Indicates high-difficulty negative samples; , represents the Euclidean distance between the anchor sample and the negative sample; Represents a hyperparameter used to set the reference distance of the operation scale, such as the distance between positive sample pairs or a preset value; represents the average training loss in the most recent training round, Indicates that it is used for adjustment Scaling factor affecting the intensity.

[0052] After obtaining the difficulty perception tuple in the metric space, the next goal is to map it back to the feature space for model training. Based on the anchor samples, positive samples, positive sample labels, high-difficulty negative samples and negative sample labels in the metric space, the generator generates enhanced data samples in the feature space, where the enhanced data samples include anchor samples, positive samples, positive sample labels, enhanced negative samples and negative sample labels. Specifically, a generator is trained. , the generator can map the enhanced embedded features in the tuple back to the original feature space , which is used in the subsequent training process. Since it is usually difficult for the generator to perfectly map all embedded features in practical applications, in order to ensure the availability and consistency of the synthesized features in the feature space, not only the enhanced negative samples are mapped, but also the remaining unchanged samples in the tuple are mapped together, thereby maintaining the consistency and trainability of the entire tuple, as shown in the following formula:

[0053] ;

[0054] in, and Represent tuples in feature space and embedding space respectively; Represents a generator; Represents the parameters of the generator.

[0055] In order to effectively utilize difficulty-aware synthetic features in training , it must ensure that it preserves both the class label of the original sample and the "difficulty" characteristics of the augmented one. This is constrained by the generation loss function, which consists of two parts: the reconstruction loss function and the softmax loss function. In order for the synthesized high-difficulty negative sample to be semantically consistent with the augmented high-difficulty negative sample, it must form a valid tuple that satisfies the specified "difficulty level". Therefore, the optimization goal of the generator is as follows:

[0056] ;

[0057] in, represents the generation loss, represents the reconstruction loss, represents the softmax loss, represents the balance factor; represents the original data sample features in the metric space (the set of feature vectors extracted from the image by the convolutional neural network), Represents the synthetic features of the original data sample features (unenhanced embedding) after the generator (generator embeds the original The result of mapping back to feature space), Represents the enhanced data sample features (the generator embeds the enhanced The result of mapping back to the feature space is the difficulty-perceived synthetic feature); represents the target category. The softmax loss is calculated as follows:

[0058] ;

[0059] in, Represents the enhanced data sample features; ,express The corresponding original data sample label (target category); Indicates the total number of categories; Indicates that Mapped to the output value of the target category; Indicates that Mapped to the sum of the output values ​​of all categories; represents the natural exponential function.

[0060] Reconstruction losses It is used to force the generator to learn a "lossless" mapping relationship by minimizing the difference between the original features and the reconstructed features, ensuring that the hardness adjustment of the embedding space (such as distance change) can be accurately reflected in the feature space. Each sample point is as close to its original position as possible during the mapping process; at the same time, the softmax loss Ensure that the synthesized enhanced features can retain the original semantic labels.

[0061] After obtaining the augmented data samples that retain the original labels, they are further used to train DHAH. The convolutional neural network is trained with the augmented data samples, and the parameters of the convolutional neural network are updated according to the target loss to obtain DHAH. The target loss function of DHAH is shown as follows:

[0062] ;

[0063] in, Indicates target loss; Represents a preset parameter. represents the triplet loss calculated on the original data sample, Denotes the triplet loss calculated for the enhanced data sample. In this embodiment, .

[0064] Introduction As a balancing factor, it is used when the generator is not fully trained, i.e., the generation loss When it is high, the weight allocation to the synthetic features should be appropriately reduced, because the synthetic features generated at this time may lack sufficient authenticity and semantic relevance. and Both are triplet loss functions, defined as:

[0065] ;

[0066] ;

[0067] in, represents the Euclidean distance, Indicates the set margin. In this embodiment, . Figure 3 Shown is a schematic representation of triplet loss.

[0068] Notably, the above process generates synthetic features that preserve both the sample's "difficulty" and semantic labels. A concurrently trained generator is then used to map the enhanced features from the embedding space back to the feature space, preserving both the labels and the enhanced difficulty information to complement the original embedded features, thereby fully utilizing the hidden information in all samples. DHAH does not directly generate discriminative hash codes that preserve semantic information after optimizing the CNN. Instead, it only obtains continuous class hash codes for different bits within the range [−1, 1]. Furthermore, test samples are fed into the trained network, and the element-wise conversion symbol sign() is used to project the continuous real-valued features into the corresponding discrete codes.

[0069] Finally, the image to be retrieved is input into the trained DHAH to achieve image retrieval.

[0070] To demonstrate the performance of the image retrieval model (DHAH) in this embodiment, experiments were conducted using the MIRFLICKR-25K, NUS-WIDE, and MS COCO datasets. The experiments used the open-source PyTorch library and were conducted on a Linux server equipped with an NVIDIA RTX 3090 GPU.

[0071] Ten typical deep cross-modal image retrieval models were selected for comparison with the image retrieval model of the present invention (abbreviated as DHAH), including Deep Supervised Hashing (DSH), HashNet, OrthoHash, Improved Deep Hashing (IDHN), Hybrid Proxy Pair Loss (HyP 2 Loss), Hash-guided Hinge Function (HHF), CenterHash, Causal Feature Learning to Binary Injection Hashing (CFBH), Deep Consistent Penalized Hashing with Noise-Robust Representation (DCPH), and Distribution-Consistency Guided Multimodal Hashing (DCGMH). The models used for comparison are based on the official source code, and the relevant parameter settings are based on the original paper.

[0072] To evaluate the model's performance in hash retrieval, we used three common evaluation metrics: mean average precision (mAP), precision-recall (PR) curve, and TopN precision curve. The mAP score represents the average accuracy of the model across the entire dataset and is a widely recognized standard performance metric in the information retrieval field. The PR curve shows the relationship between recall and precision, providing insight into overall retrieval performance. The TopN precision curve measures accuracy based on the top-N samples returned.

[0073] The mean average precision (mAP) performance of different models on the MIRFLICKR-25K, NUS-WIDE, and MS COCO datasets are shown in Tables 1-3. It can be seen that DHAH is significantly better than other models.

[0074] Table 1 mAP of different models on MIRFLICKR-25K dataset with 16, 32, 64 and 128 bits

[0075]

[0076] In Table 1, MIRFLICKR-25K@all indicates sorting based on all data in the MIRFLICKR-25K library.

[0077] Table 2 mAP of different models on the NUS-WIDE dataset with 16-bit, 32-bit, 64-bit and 128-bit

[0078]

[0079] In Table 2, NUS-WIDE@5000 indicates the ranking based on 5000 data in the NUS-WIDE library.

[0080] Table 3 mAP of different models on the MS COCO dataset with 16-bit, 32-bit, 64-bit and 128-bit

[0081]

[0082] In Table 3, MS COCO@all indicates sorting based on all data in the MS COCO library.

[0083] In addition to the mAP comparison, the effectiveness of DHAH is further evaluated by plotting the 32-bit precision-recall (PR) curve and the TopN accuracy curve, as shown in Figure 2. Figure 4-Figure 6 shown.

[0084] To evaluate the effectiveness of each module in DHAH, ablation experiments were conducted. Three variants were designed: DPSH-pairwise, DSH-triplewise, and DHAH-pairwise, and a simplified study was conducted on the MIRFLICKR-25K dataset. Each variant modifies the original DHAH framework. Specifically, DPSH-pairwise replaces the objective loss function in DHAH with the pairwise loss function used in DPSH (Li WJ, Wang S, Kang W C. Feature learning based deepsupervised hashing with pairwise labels[J]. arXiv preprint arXiv:1511.03855,2015.); DSH-triplewise replaces the objective loss function in DHAH with the triplet loss function in DSH (LiuH, Wang R, Shan S, et al. Deep supervised hashing for fast image retrieval[C] / / Proceedings of the IEEE conference on computer vision and patternrecognition. 2016: 2064-2072.); DHAH-pairwise replaces the triplet-based loss in DHAH with a pairwise loss function. Table 4 shows the comparison of the mean average precision (mAP) results of these variants with the original DHAH. It can be seen that the performance of each variant is lower than the full DHAH architecture. It is worth noting that DHAH outperforms DSH-triplewise, while DHAH-pairwise outperforms DPSH-pairwise, which proves the effectiveness of the objective loss function in DHAH.

[0085] Table 4. mAP of DHAH and its variants on MIRFLICKR-25K dataset with 16, 32, 64 and 128 bits

[0086]

[0087] In summary, DHAH achieves excellent image retrieval performance, attributed to the introduction of the adaptive difficulty-aware enhancement module, which strengthens representation learning. Furthermore, the difficulty-aware feature synthesis module generates synthetic data with preserved labels for repeated training. Furthermore, the objective loss function facilitates model optimization. This allows the model to fully exploit the potential information contained in all samples, achieving optimal performance.

[0088] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form or substance. It should be pointed out that ordinary technicians in this technical field can make several improvements and supplements without departing from the method of the present invention. These improvements and supplements should also be regarded as the scope of protection of the present invention. Any equivalent changes, modifications and evolutions made by technicians familiar with this profession without departing from the spirit and scope of the present invention by using the technical content disclosed above are all equivalent embodiments of the present invention; at the same time, any equivalent changes, modifications and evolutions made to the above embodiments based on the essential technology of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for training an image retrieval model based on deep difficulty-aware hashing, characterized in that: The following steps are involved: Obtaining original data samples, where the original data samples include anchor samples, positive samples, positive sample labels, negative samples, and negative sample labels; Mapping the original data samples from feature space to metric space; In the metric space, based on the anchor samples and negative samples, high-difficulty negative samples are generated, where the distance between the high-difficulty negative samples and the anchor samples is smaller than the distance between the negative samples and the anchor samples; Based on the anchor samples, positive samples, positive sample labels, high-difficulty negative samples and negative sample labels in the metric space, the generator generates enhanced data samples in the feature space, where the enhanced data samples include anchor samples, positive samples, positive sample labels, enhanced negative samples and negative sample labels; The convolutional neural network is trained by augmenting data samples, and the parameters of the convolutional neural network are updated according to the target loss to obtain an image retrieval model. Among them, the high-difficulty negative samples are calculated by the following formula: ； in, represents the anchor point sample, represents negative samples, Indicates high-difficulty negative samples; , represents the Euclidean distance between the anchor sample and the negative sample; represents a hyperparameter; represents the average training loss in the most recent training round, Indicates that it is used for adjustment Scaling factors that affect intensity; Update the parameters of the generator according to the generation loss, where the generation loss is calculated as follows: ； in, represents the generation loss, represents the reconstruction loss, represents the softmax loss, represents the balance factor; Represents the original data sample features in the metric space; Represents the synthetic features of the original data sample features after the generator; Represents the enhanced data sample features; Indicates the target category.

2. The method according to claim 1, characterized in that The softmax loss is calculated as follows: ； in, Represents the enhanced data sample features; ,express The corresponding original data sample label; Indicates the total number of categories; Indicates that Mapped to the output value of the target category; Indicates that Mapped to the sum of the output values ​​of all categories; represents the natural exponential function.

3. The method according to claim 1, characterized in that The target loss is calculated as follows: ； in, Indicates target loss; Represents a preset parameter. represents the triplet loss calculated on the original data sample, represents the triplet loss calculated on the augmented data samples.

4. The method according to claim 3, characterized in that Calculated by the following formula: ； in, represents the Euclidean distance, Indicates the set interval boundary; represents the anchor point sample, represents a positive sample, represents negative samples.

5. The method according to claim 3, characterized in that Calculated by the following formula: ； in, represents the Euclidean distance, Indicates the set interval boundary; represents the anchor point sample, represents a positive sample, Represents high-difficulty negative samples.

6. An image retrieval method based on deep difficulty-aware hashing, characterized in that The following steps are involved: The image to be retrieved is input into the image retrieval model obtained by the image retrieval model training method based on deep difficulty perceptual hashing described in any one of claims 1-5, and the image retrieval model outputs the retrieval result.

7. An electronic device, characterized in that: It includes a memory and a processor, wherein the memory is used to store a computer program; the processor is used to execute the image retrieval model training method based on deep difficulty perceptual hashing as described in any one of claims 1 to 5 or the image retrieval method based on deep difficulty perceptual hashing as described in claim 6 according to the computer program.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program; when the computer program is executed by a processor, it implements the image retrieval model training method based on deep difficulty-perceptual hashing described in any one of claims 1 to 5 or the image retrieval method based on deep difficulty-perceptual hashing described in claim 6.

Images