Cross-batch mining-based image retrieval model training method and image retrieval method

Abstract

The invention relates to the technical field of information retrieval, in particular to an image retrieval model training method based on cross-batch mining and an image retrieval method. According to the method, the phenomenon of slow drifting of the features is found, a memory enhancement strategy is designed based on the phenomenon, and the deep features of the small batches are recorded and updated through a cross-batch memory module, so that the image retrieval model can mine enough information negative samples across multiple small batches.

Description

Technical Field

[0001] This invention relates to the field of information retrieval technology, specifically to a method for training an image retrieval model based on cross-batch mining and an image retrieval method. Background Technology

[0002] In today's digital age, the widespread adoption of mobile devices and the rapid development of internet technology have spurred an explosive growth in high-dimensional and complex multimedia information, posing numerous serious challenges to multimedia retrieval technology. How to accurately and efficiently retrieve the information users need from massive amounts of multimedia data is an urgent problem to be solved. Approximate Nearest Neighbor (ANN) algorithms are widely used in recommender systems, natural language processing, and other fields. Among them, hashing technology has become a research hotspot due to its efficiency and storage advantages. It aims to project high-dimensional features onto a low-dimensional Hamming space while preserving semantic similarity, and returns a ranked sample list by calculating the Hamming distance through XOR operations.

[0003] Hash learning methods are mainly divided into shallow hashing and deep hashing. Shallow hashing has a simple structure but limited ability to model complex data, and performs poorly on large-scale complex datasets. Deep hashing combines deep learning and hash learning to build an end-to-end model, significantly improving the model's robustness and retrieval performance.

[0004] Pair-wise hashing algorithms (such as contrastive loss, triplet loss, and multiple similarity loss) based on mini-batch data to construct pairwise similarity are widely used in hash retrieval. Most of them can be unified into a weighting scheme within the General Pair Weighting (GPW) framework, and their performance largely depends on the ability to mine effective negative samples. Current research mainly obtains effective negative samples by sampling based on the overall data distribution or by weighting samples within mini-batch data. However, the ability to mine negative samples is limited by the size of the mini-batch data. Increasing the size of the mini-batch data has limitations such as GPU memory, cost, and cross-device synchronous training. Furthermore, methods that screen difficult negative samples from the entire dataset are not suitable for large-scale datasets. Summary of the Invention

[0005] To address at least one problem in the prior art, the present invention provides an image retrieval model training method and an image retrieval method based on cross-batch mining.

[0006] The first aspect of this invention provides a method for training an image retrieval model based on cross-batch mining, which includes the following steps: Construct an image retrieval model that includes cross-batch memory modules; Extract the current embedding features of a mini-batches of image samples, and update the cross-batch memory module based on the current embedding features. Specifically, add the current embedding features to the embedding feature queue of the cross-batch memory module, and remove the oldest embedding feature from the embedding feature queue of the cross-batch memory module. Calculate the similarity matrix between the current embedded feature and the embedded features across batch memory modules. Based on the similarity matrix Calculate the target loss and update the parameters of the image retrieval model based on the target loss to obtain the trained image retrieval model.

[0007] In this invention, the phenomenon of "slow feature drift" was discovered. Based on this phenomenon, a memory enhancement strategy was designed. By recording and updating the deep features of small batches through cross-batch memory modules, the image retrieval model can mine sufficient informative negative samples across multiple small batches.

[0008] In some optional embodiments, the target loss is calculated by the following formula: ; in, , representing the similarity matrix; and These represent the currently embedded feature and the embedded feature across batch memory modules, respectively. and These represent the size of the small batch and the cross-batch memory modules, respectively; Indicates assignment to The weight.

[0009] In some optional embodiments, a step of pre-training the image retrieval model is included before updating the cross-batch memory modules based on the current embedded features.

[0010] In some optional embodiments, the warm-up training consists of 10 rounds.

[0011] In some optional embodiments, during warm-up training, the similarity matrix is ​​calculated only based on the current mini-batch of image samples. And based on the similarity matrix Calculate the target loss.

[0012] In some optional embodiments, the image retrieval model includes a feature extraction module and a hash learning module, wherein the hash learning module includes a cross-batch memory module.

[0013] In some optional embodiments, the feature extraction module is a ResNet-50 network.

[0014] A second aspect of the present invention provides an image retrieval method based on cross-batch mining, comprising the following steps: The image to be retrieved is input into the image retrieval model, and the image retrieval model outputs the retrieval results. The image retrieval model is obtained by the image retrieval model training method based on cross-batch mining described above.

[0015] A third aspect of the present invention provides an electronic device comprising a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the above-described image retrieval method based on cross-batch mining according to the computer program.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium for storing a computer program; the computer program, when executed by a processor, implements the above-described image retrieval method based on cross-batch mining.

[0017] The technical solution of this invention has the following beneficial effects: by designing an embedded feature queue, the current embedded feature is added to the embedded feature queue in each iteration, and the earliest embedded feature in the embedded feature queue is removed, thereby recording and updating the deep features of small batches, so that the image retrieval model can mine negative instances with sufficient information across multiple small batches, which is suitable for large-scale image retrieval. Attached Figure Description

[0018] Figure 1 This is a verification diagram of the feature drift phenomenon in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the network structure of the image retrieval model in an embodiment of the present invention.

[0020] Figure 3 The performance curves of different models on the CIFAR-10 dataset are shown, where (a) is the accuracy-recall curve and (b) is the Top N accuracy curve.

[0021] Figure 4 The performance curves of different models on the MIRFLICKR25K dataset are shown, where (a) is the accuracy-recall curve and (b) is the Top N accuracy curve.

[0022] Figure 5 The performance curves of different models on the MSCOCO dataset are shown, where (a) is the accuracy-recall curve and (b) is the Top N accuracy curve. Detailed Implementation

[0023] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0024] Existing hash retrieval methods offer various complex schemes, but their negative sample mining capabilities are fundamentally limited by the size of mini-batches of data, which directly affects the number of available training sample pairs. Since the number of negative sample pairs increases quadratically with the size of the mini-batches, directly increasing the mini-batch size can quickly improve the performance of pairwise hashing algorithms and enhance sampling effectiveness. However, this is not the key to solving the problem of difficult negative sample mining because it has two limitations: first, it is limited by GPU memory capacity and cost; second, large-scale mini-batch data requires synchronous training across devices, which is extremely difficult to implement.

[0025] During the training of an image retrieval model, the model parameters constantly change, causing embedding features from previous small batches of data to be considered "outdated features." Typically, these "outdated features" are discarded. However, the inventors discovered that these "outdated features" are actually important resources, and utilizing them requires no additional computational cost. The inventors further analyzed the feature differences of the same image samples in different training iterations, discovering a phenomenon called "slow feature drift." For input image samples... In the In the next iteration, the step size is The characteristic drift is shown below: ; in, Indicates the first Model parameters for each training iteration Indicates compared to the first Early iteration Model parameters at each iteration step. This represents a feature extraction network.

[0026] The model was trained from scratch on the MIRFLICKR-25K dataset using triplet loss. For a randomly selected set of samples, the average feature shift was calculated at different training steps {10, 100, 1000}. The results are as follows: Figure 1As shown, when the training steps are 10 and 100, the feature shift remains at a relatively small level; however, when the training steps are 1000, the embedded features change drastically in the first 1250 iterations, and then gradually stabilize in subsequent iterations. In other words, the phenomenon of "slow feature drift" is visually verified.

[0027] The "slow feature drift" phenomenon can establish an upper bound for the gradient error of pairwise hashing algorithms. Assume the contrastive loss of a single negative sample pair... ,in and These represent the embedding vectors respectively; the exact loss gradient is shown below: ; in, Indicates loss For model parameters gradient, Representing embedded features For model parameters The gradient.

[0028] The gradient of the approximate loss is shown below: ; Assumption , ,in It is an embedded feature Approximate value, network structure If the Lipschitz continuity condition is satisfied, the gradient error between the exact loss gradient and the approximate loss gradient is as follows:

[0029] in, Here, is the Lipschitz constant. The above formula shows that, under the Lipschitz assumption, the gradient error is controlled by the sample embedding error. Therefore, the "slow feature drift" phenomenon ensures that effective negative samples can be provided for pairwise hashing algorithms when mining negative samples across multiple mini-batch datasets.

[0030] Based on this, the present invention proposes an image retrieval model training method and an image retrieval method based on cross-batch mining. Specifically, an image retrieval model is trained and then used to perform image retrieval. For ease of representation, the image retrieval model in this embodiment is labeled DANSH.

[0031] Training an image retrieval model using an image dataset, assuming a training dataset... Include Image samples, of which Indicates the first There are N image samples, where N represents a positive integer. This represents manually labeled information, where each label is encoded in One-Hot format; Indicates the first The sample belongs to the first Categories Then it means the first The sample does not belong to the first There are several categories.

[0032] like Figure 2 As shown, the image retrieval model includes a feature extraction module and a hash learning module.

[0033] The feature extraction module can use the classic ResNet-50 network to efficiently capture the embedded features of the samples. Specifically, for any input image sample... Its embedding features are calculated using a ResNet-50 network and can be represented as follows: ,in This represents the network structure of ResNet-50 (including the forward computation logic for feature extraction). This represents the model parameters learned during network training (such as convolutional layer weights, fully connected layer biases, etc.).

[0034] Output of feature extraction module Let K be a continuous vector of K bits (K being a positive integer). During the model validation phase, to convert the continuous vector into hash code, a bit-based transformation function is used through a hash layer to generate hash code from the continuous vector, as shown in the following equation: ; in, Represents binary encoding. This represents the output of the feature extraction module; if ,So ,otherwise .

[0035] In the hash learning module, the embedded features output from the feature extraction module are first fed into the standard triplet loss function, and the similarity matrix of the current embedded samples is calculated. This guides model training optimization. A warm-up mechanism is implemented during training: when the number of training epochs exceeds a predefined warm-up number (e.g., 10 epochs), the embedded features and their corresponding sample labels generated in the current epoch are stored in a cross-batch memory module. In subsequent training epochs, the embedded features and their corresponding sample labels from each epoch are sequentially added to this cross-batch memory module. When the amount of data stored in the cross-batch memory module reaches its preset maximum capacity, a rolling update mechanism is initiated. Specifically, the rolling update mechanism of the cross-batch memory module moves the memory pointer forward by the length of a small batch, dequeues the earliest set of embedded features and labels, and simultaneously enqueues the newly captured features and their corresponding sample labels. This ensures that the cross-batch memory module always stores the latest cross-batch data that meets the capacity limit, providing sufficient historical embedded features and labels that approximate the current feature distribution for subsequent training, supporting the efficient mining of informative negative samples from cross-batch memory.

[0036] When the number of training epochs exceeds the predefined number of warm-up epochs, the calculation method for the standard triplet loss is adjusted: instead of calculating the similarity matrix based solely on samples within the current mini-batch, historical batch embedding features stored in cross-batch memory modules are introduced to construct an approximate similarity matrix. This is to uncover more informational negative samples. Specifically, for the current small batch of the first... 1 sample (embedded features are) ) and the first in the cross-batch memory module historical samples (embedded features are) The inner product of the two vectors (since the features have been normalized, the inner product result is equivalent to cosine similarity) is used to obtain the result. The Each element, namely The final dimension is "current mini-batch size". × Memory size of memory modules across batches Approximate similarity matrix of " Based on this matrix, the standard triplet loss can mine informative negative samples across current and historical batches, thereby optimizing image retrieval model parameters and improving model performance.

[0037] Overall, given the significant feature drift during the initial training phase of the model, a warmup of the network architecture was performed to allow the model to enter a region where the embedded feature changes are stable. Subsequently, the warmed-up model was used to compute features from random mini-batch data, thereby initializing the cross-batch memory module. , ,in and Initialized to the first Sample Embedded features and corresponding labels, Indicates the size of memory modules across batches. Definition Memory percentage represents the ratio of the size of the memory module to the size of the training samples.

[0038] The cross-batch memory module is updated and maintained as a queue. In each iteration, the sample embeddings and corresponding labels of the current mini-batch data are added to the queue, while the earliest embedded features and corresponding labels in the queue are dequeued. By repeating this process, the cross-batch memory module can be updated with the embeddings of the latest mini-batch data without incurring additional overhead. It is worth noting that since the memory required to store feature embeddings is minimal, the feature embeddings of the entire training set can be cached in the cross-batch memory module during model training.

[0039] Based on the General Pair Weighting framework, pairwise hashing algorithms can be transformed into a unified weighted form of sample pair similarity within mini-batch data. During the warm-up phase of model training, because the model parameters are not yet stable and the embedded features have significant drift, the cross-batch memory module is not enabled; the aforementioned similarity matrix is ​​calculated only within the current mini-batch data. This is to achieve initial convergence of the model parameters; during the warm-up training phase, the cross-batch memory module dynamically maintains the embedding features and corresponding labels of previous mini-batches in the form of a queue, and only needs to calculate the embedding features of the current mini-batch samples. Historical embedding features stored in the queue cross-batch similarity matrix Formally, the pairwise target loss for memory augmentation is shown in the following formula: ; in, Indicates the first in the current small batch The cross-batch loss term corresponding to each sample (as the anchor sample); , representing the similarity matrix; Indicates the size of the small batch; Indicates the size of memory modules across batches; Indicates assignment to The weights. For negative sample pairs in the triplet loss, ,in This represents the set of valid positive samples that share the same anchor point. , , and These represent the indices of the anchor sample, negative sample, and positive sample, respectively. This represents a predefined threshold for the triplet loss. The total loss for the entire batch. By all Anchor point sample By averaging, this design allows the model to utilize massive historical samples across batches of memory to mine more informational negative samples, thereby improving the performance of image retrieval tasks.

[0040] By calculating a new similarity matrix The above formula is a weighted scheme for memory-enhanced pairwise hash functions. Samples in the current mini-batch data are compared with samples in the memory module to extract valuable information for model training from negative instances using the pairwise hash algorithm. Specifically... loss function The gradient of is shown in the following formula: .

[0041] Furthermore, the chain rule can be applied to calculate the model parameters. The relevant gradient is shown in the following formula: ; Finally, the model parameters were optimized using stochastic gradient descent. .

[0042] To verify the effectiveness of the image retrieval model (DANsH) in this embodiment of the invention, performance verification experiments were conducted on three publicly available benchmark datasets: CIFAR-10, MIRFLICKR-25K, and MSCOCO. Furthermore, the construction of DANsH and all experiments were implemented based on the open-source deep learning framework PyTorch and deployed on a Linux server equipped with an NVIDIA GeForce RTX 4090 GPU.

[0043] The CIFAR-10 dataset is a classic dataset in the field of image recognition and retrieval, containing 60,000 color photographs divided into 10 different categories. The images have a resolution of 32×32 pixels and have blue, red, and green channels. In the experiments, 4,000 images were randomly selected as the training set, 1,000 as the query set, and the remaining images formed the database.

[0044] The MIRFLICKR-25K dataset is widely used in the field of multimedia visual retrieval, containing 25,000 images covering 38 categories. In the experiments, 4,000 images were randomly selected as the training set, and another 1,000 images were used as the query set.

[0045] The MSCOCO dataset contains over 120,000 images across 80 categories. In the experiments, unlabeled samples were first removed. Then, a query set of 5,000 images and a training set of 10,000 images were randomly selected.

[0046] In addition, 10 typical deep cross-modal image retrieval models were selected as comparison models, including Deep Supervised Hash (DSH), HashNet, OrthoHash, Improved Deep Hash (IDHN), and Hybrid Proximity Pair Loss (HyP). 2 The comparison models include Loss, Hash Guided Hinge Function (HHF), CenterHash, Causal Feature Learning Hash (CFBH), Deep Semantic Consistency Penalized Hash with Noise Robust Representation (DCPH), and Distribution Consistency Guided Multimodal Hash (DCGMH). All comparison models are built based on their official source code, and the relevant parameter settings during model training and testing are performed in accordance with the configuration schemes in the original papers corresponding to each model.

[0047] The analysis was conducted using three mainstream evaluation metrics: mean precision (mAP), precision-recall (PR) curve, and TopN precision curve.

[0048] As shown in Tables 1-3, DANsH demonstrates excellent retrieval performance on three public benchmark datasets: CIFAR-10, MIRFLICKR-25K, and MSCOCO.

[0049] Table 1. mAP of different models on the CIFAR-10 dataset (@all indicates that all data in the dataset is sorted)

[0050] Table 2. mAP of different models on the MIRFLICKR-25K dataset (@all indicates that all data in the dataset is sorted).

[0051] Table 3. mAP of different models on the MS COCO dataset (@all indicates that all data in the dataset is sorted)

[0052] In addition to comparing the mAP metric, the effectiveness of the DANSH model was further evaluated from more dimensions by plotting a 32-bit precision-recall (PR) curve and a Top-N precision curve. The relevant results are as follows: Figures 3 to 5 As shown.

[0053] To verify the general effectiveness of the cross-batch memory module for pair-based loss, variant experiments were designed on the CIFAR-10 and MIRFLICKR-25K datasets. The experimental settings are summarized in Tables 4 and 5. The symbol "√" indicates that the corresponding module was used. The experiments show that when the three pair-based loss functions are combined with the cross-batch memory module, their performance is improved, fully validating the effectiveness of the cross-batch memory module.

[0054] Table 4. Performance impact of cross-batch memory modules on pair-based loss on the CIFAR-10 dataset.

[0055] Table 5. Performance impact of cross-batch memory modules on pair-based loss on the MIRFLICKR-25K dataset.

[0056] In summary, DANsH demonstrates outstanding image retrieval performance, and the core reason for this superior performance lies in its cross-batch memory module. This module stores and retrieves embedded features in mini-batches, providing ample informative negative samples for pairwise hashing algorithms. This, in turn, drives the model to mine more discriminative negative samples, ultimately ensuring its performance advantage.

[0057] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any form or substance. It should be noted that those skilled in the art can make various improvements and additions without departing from the method of the present invention, and these improvements and additions should also be considered within the scope of protection of the present invention. Any modifications, alterations, and equivalent changes made by those skilled in the art based on the above-disclosed technical content without departing from the spirit and scope of the present invention are equivalent embodiments of the present invention. Furthermore, any modifications, alterations, and evolutions made to the above embodiments based on the essential technology of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A method for training an image retrieval model based on cross-batch mining, characterized in that, Includes the following steps: Construct an image retrieval model that includes cross-batch memory modules; Extract the current embedding features of a small batch of image samples, and update the cross-batch memory module based on the current embedding features. Specifically, add the current embedding features to the embedding feature queue of the cross-batch memory module, and remove the earliest embedding feature from the embedding feature queue of the cross-batch memory module. Calculate the similarity matrix between the current embedded feature and the embedded features across batch memory modules. Based on the similarity matrix Calculate the target loss and update the parameters of the image retrieval model based on the target loss to obtain the trained image retrieval model.

2. The method according to claim 1, characterized in that, The target loss is calculated by the following formula: ； in, , representing the similarity matrix; and These represent the embedding vectors respectively; and These represent the size of the small batch and the cross-batch memory modules, respectively; Indicates assignment to The weight.

3. The method according to claim 1, characterized in that, Before updating the cross-batch memory modules based on the current embedded features, a pre-training step for the image retrieval model is also included.

4. The method according to claim 3, characterized in that, The warm-up training consists of 10 rounds.

5. The method according to claim 3, characterized in that, During warm-up training, the similarity matrix is ​​calculated only based on the current mini-batch of image samples. And based on the similarity matrix Calculate the target loss.

6. The method according to claim 1, characterized in that, The image retrieval model includes a feature extraction module and a hash learning module, wherein the hash learning module includes a cross-batch memory module.

7. The method according to claim 6, characterized in that, The feature extraction module is a ResNet-50 network.

8. An image retrieval method based on cross-batch mining, characterized in that, Includes the following steps: The image to be retrieved is input into the image retrieval model, and the image retrieval model outputs the retrieval results; The image retrieval model is obtained by the image retrieval model training method based on cross-batch mining as described in any of claims 1-7.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the image retrieval method based on cross-batch mining as described in any one of claims 1-7 according to the computer program.

10. A computer-readable storage medium, characterized in that, A computer-readable storage medium is used to store a computer program that, when executed by a processor, implements the image retrieval method based on cross-batch mining as described in any one of claims 1-7.

Images