A training method for an image retrieval model and an image retrieval algorithm based thereon.

Abstract

This invention proposes a training method for an image retrieval model. The method includes: S1, acquiring a training set and a query image, wherein the training set includes multiple positive sample images of the same class as the query image and multiple negative sample images of different classes; S2, using the training set and query images obtained in step S1 to train the image retrieval model multiple times until convergence, and updating the image retrieval model parameters using gradient based on the AUPRC target loss during the training process. This invention constructs an upper bound of AUPRC as the substitution loss (the substitution loss is the AUPRC target loss) and introduces an auxiliary vector for positive sample image estimation in the random estimation, thereby achieving the generalizability and asymptotic unbiasedness of the substitution target.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically to the field of image retrieval within computer vision, and more specifically, to a training method for an image retrieval model and an image retrieval algorithm based thereon. Background Technology

[0002] With the rapid expansion of internet information, extracting desired information from massive amounts of data, especially from unstructured information, has become a challenging task. Image retrieval tasks, widely used in search engines, intelligent security, and risk content monitoring, require searching for candidate images containing the same or similar content from massive image databases (e.g., images in image databases) and ranking these candidate images according to their similarity to the query image.

[0003] In recent years, with the rapid development of deep learning, mainstream image retrieval technologies are based on deep models (such as image retrieval models) and constrain the similarity of relevant images (images related to the image retrieval task) to be higher than that of irrelevant images (images unrelated to the image retrieval task) by comparing the loss of the deep model's output results. This allows the deep model to have image retrieval capabilities. However, such image retrieval techniques ignore the overall ranking of candidate images composed of relevant images. They cannot further verify and process the ranking results according to the needs of real-world scenarios, nor can they utilize the top-ranked parts of the overall ranking to improve the performance of images with significant practical implications. To improve the image retrieval performance focusing on the top images in image retrieval technologies, existing techniques directly optimize the Area Under the Precision-Recall Curve (AUPRC) to enhance the overall retrieval performance of the model. Among these, the unbiased estimate Average Precision (AP) in AUPRC is one of the standard metrics for image retrieval, aiming to measure the balance between precision and recall of an image retrieval model under different thresholds. Therefore, by directly optimizing AUPRC, a larger loss can be assigned to misclassified images that rank highly, making the image retrieval model more in line with the needs of real-world scenarios.

[0004] However, existing image retrieval methods based on AUPRC suffer from a mismatch between the training objective and the test metric of the retrieval model. That is, during training, pairwise loss or triplet loss is usually used to adjust the parameters of the retrieval model, which cannot fully optimize the standard metric AUPRC, thus becoming one of the bottlenecks in the development of deep image retrieval technology. Thanks to the excellent properties and wide application prospects of AUPRC, the optimization problem of deep image retrieval technology has attracted widespread attention from academia and industry in order to unify the training objective and test metric of the retrieval model in the learning framework, and related theories and technologies have made significant progress. Given that AUPRC requires sorting all samples, early methods, such as the schemes in references [1], [2], and [3], mainly used full batch training. These methods require a lot of computing resources and cannot adapt to the ever-increasing data and model scale in the era of deep learning. Therefore, recent related studies, such as the schemes in references [4], [5], and [6], all aim to focus on solving the AUPRC stochastic optimization problem. However, the existence of non-differentiable objective functions in these related studies makes gradient-based model optimization methods infeasible.

[0005] Existing methods address the issue of non-differentiability of the objective function by proposing solutions to replace the non-differentiable 0-1 loss in the objective function with a differentiable alternative loss. Examples include the exponential loss used in reference [7], the sigmoid loss used in reference [8], and the linear interpolation function used in reference [9]. These solutions aim to address the problem that the objective function is non-differentiable due to some losses, thus preventing the updating of model parameters based on gradients. Although these works avoid the problem of non-differentiability of the objective function, the relationship between the objective function after using the alternative loss and the original objective function is difficult to guarantee, and cannot effectively support model algorithm design and theoretical analysis.

[0006] The references are as follows:

[0007] [1]Donald Metzler and W Bruce Croft.A markov random field model forterm dependencies.In international ACM SIGIRConference on Research and Development in Information Retrieval, pages 472-479, 2005.

[0008] [2]Pritish Mohapatra,CV Jawahar,and M Pawan Kumar.Efficientoptimization for average precision svm.Advances in Neural lnformationProcessing Systems,27:2312-2320,2014.

[0009] [3]Mark Goadrich,Louis Oliphant,and Jude Shavlik.Gleaner:Creatingensembles of first-order clauses to improve recal I-precision curves.MachineLearning,64(1-3):231-261,2006.

[0010] [4]Andrew Brown,Weidi Xie,Vicky Kalogeiton,and AndrewZisserman.Smooth-ap:Smoothing the path towards large-scale image retrieval.InEuropean Conference on Computer Vision,pages677-694.Springer,2020.

[0011] [5]Fatih Cakir,Kun He,Xide Xia,Brian Kulis,and Stan Sclaroff.Deepmetric learning to rank.In lEEE / CVF Conference on Computer Vision and PatternRecognition,pages 1861-1870,2019.

[0012] [6]Paul Henderson and Vittorio Ferrari.End-to-end training of objectclass detectors for mean average precision.In Asian Conference on ComputerVision,pages 198-213.Springer,2016.

[0013] [7] Tao Qin, Xu-Dong Zhang, Ming-Feng Tsai, De-Sheng Wang, Tie-Yan Liu, and Hang Li. Query-level loss functions for information retrieval. Information Processing & Management, 44(2): 838-855, 2008.

[0014] [8] Andrew Brown, Weidi Xie, Vicky Kalogeiton, and Andrew Zisserman. Smooth-ap: Smoothing the path towards large-scale image retrieval. InEuropean Conference on Computer Vision, pages 677-694. Springer, 2020.

[0015] [9] Qijia Jiang, Olaoluwa Adigun, Harikrishna Narasimhan, Mahdi MilaniFard, and Maya Gupta. Optimizing black-box metrics with adaptive surrogates. InInternational Conference on Machine Learning, pages 4784-4793. PMLR, 2020. Summary of the Invention

[0016] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a training method for an image retrieval model and an image retrieval algorithm based thereon.

[0017] The objective of this invention is achieved through the following technical solution:

[0018] According to a first aspect of the present invention, a training method for an image retrieval model is provided, the method comprising: S1, acquiring a training set and a query image, wherein the training set includes multiple positive sample images of the same class as the query image and multiple negative sample images of different classes from the query image; S2, using the training set and query image obtained in step S1 to train the image retrieval model for multiple rounds until convergence, and updating the parameters of the image retrieval model in a gradient manner based on the AUPRC target loss during the training process.

[0019] In some embodiments of the present invention, the AUPRC target loss is:

[0020]

[0021] Where w represents the parameters of the image retrieval model, S represents the set of all samples in the training set, and x + S represents a positive sample image. + x represents all positive sample images in S + The set of positive samples constituted For each positive sample image x in S+, + The corresponding objective function σ(*) values ​​are averaged, where π represents the prior probability obtained beforehand, and h w (x + ) th The similarity threshold for classifying a sample image as a positive sample image is represented by l1, where l1 represents the similarity threshold given h. w (x + ) th The negative sample loss is calculated using a monotonically decreasing convex function that satisfies the Lipshitz continuity condition, where l1≥0 and l1(0)=1. Indicates a given h w (x + ) th The false positive rate corresponding to the training set at time, l2 represents the given h w (x + ) th The positive sample loss is calculated using a monotonically decreasing function that satisfies the Lipshitz continuity condition, where l2(0) = 0 and l2 ≤ 1. Indicates a given h w (x + The true positive rate corresponding to the training set at that time.

[0022] In some embodiments of the present invention, any one of the following monotonically decreasing convex functions is used to calculate the negative sample loss: smoothing function, hinge loss function, and one-sided Huber function.

[0023] Preferably, the negative sample loss is calculated using a one-sided Huber function, and the negative sample loss calculated based on the one-sided Huber function is as follows:

[0024]

[0025]

[0026] Among them, h w (x + ) th h represents the similarity threshold by which a sample image is judged as a positive sample image. w (x- ) represents the similarity score between the negative sample image predicted by the image retrieval model and the query image, and τ1 represents the first hyperparameter.

[0027] In some embodiments of the present invention, the positive sample loss is:

[0028]

[0029]

[0030] Among them, h w (x + ) th h represents the similarity threshold by which a sample image is judged as a positive sample image. w (x + ) represents the similarity score between the positive sample image predicted by the image retrieval model and the query image, and τ2 represents the second hyperparameter.

[0031] In some embodiments of the present invention, the positive sample loss is:

[0032]

[0033]

[0034] Among them, h w (x + ) th h represents the similarity threshold by which a sample image is judged as a positive sample image. w (x + ) represents the similarity score between the positive sample image predicted by the image retrieval model and the query image, and τ2 represents the second hyperparameter.

[0035] In some embodiments of the present invention, in step S2, the training set obtained in step S1 is sampled multiple times, and a batch sample set corresponding to each round of sampling is obtained. The batch sample set corresponding to each round is used to train the image retrieval model, and the image retrieval model parameters are updated using a gradient method based on the AUPRC target loss of the batch sample set corresponding to each round of training. The AUPRC target loss of the batch sample set in each round of training is:

[0036]

[0037] Where z represents the batch sample set, z + Represents all positive sample images x in z + The set that is formed This represents the average value of the objective function σ(*) for each positive sample image x+ within the set z+. Indicates that given hw (x + ) th The false positive rate corresponding to time z Indicates that given h w (x + ) th The true positive rate corresponding to time z, l1′ represents the given h w (x + ) th The negative sample loss corresponding to time z, l2′ represents the loss given h. w (x + ) th The positive sample loss corresponding to time z.

[0038] In some embodiments of the present invention, when calculating the loss of a batch sample set, an auxiliary vector is introduced and the AUPRC target loss is calculated as follows:

[0039]

[0040] in,

[0041]

[0042] in, This represents an auxiliary vector estimated using linear interpolation and exponentially weighted average of the similarity scores between the positive sample images corresponding to the batch sample set and the query image. express The number of interpolations in Indicates the introduction of auxiliary vectors The unbiased true positive rate corresponding to the batch sample set z, l2″ represents the given h w (x + ) th And sometimes introduced The loss of the positive samples afterwards.

[0043] In some embodiments of the present invention, when a positive sample auxiliary vector is introduced to increase the number of positive samples in a batch, the auxiliary vector is determined in the following manner:

[0044]

[0045] in, Let β represent the auxiliary vector corresponding to the t-th training round. j Let β represent the preset learning rate for the j-th training round. s This represents the preset learning rate corresponding to the s-th training round. This represents the linear interpolation corresponding to the similarity between positive samples and the preset query image in the s-th round of training. The similarity score between the positive sample and the preset query image in the s-th round of training.

[0046] In some embodiments of the present invention, in step S2, a semi-variance regularization loss is further introduced, and the model parameters are updated using a gradient based on the AUPRC target loss and the semi-variance regularization loss, wherein the semi-variance regularization term is a loss:

[0047]

[0048] Where λ1 is the third hyperparameter, n + h represents the number of positive sample images in the batch sample set z. w (x) represents the similarity score between the sample image in z predicted by the image retrieval model and the query image, μ + λ² represents the average similarity score between all positive sample images in z and the query image, λ² is the fourth hyperparameter, and n - z represents the number of negative sample images in z. - Represents all negative sample images x in z. - The set composed of μ - This represents the average of all similarity scores between the negative sample images in z and the query image.

[0049] In some embodiments of the present invention, τ1∈(0.01, 0.1), τ2=0.001, λ1=1, λ2=0.1.

[0050] According to a second aspect of the present invention, the present invention also provides an image retrieval algorithm, the image retrieval algorithm comprising: T1, acquiring a query image; T2, obtaining retrieval results of the query image in an image database using an image retrieval model obtained by the method described in the first aspect of the present invention.

[0051] Compared with existing technologies, the advantages of this invention are: to ensure that the relationship between the alternative optimization target and the original target is controllable, this invention proposes to... and Choose different alternative losses l1, l2, where l1 represents the loss given h. w (x + ) th The negative sample loss is calculated using a monotonically decreasing convex function that satisfies the Lipshitz continuity condition, where l2 represents the loss given h. w (x + ) thThe positive sample loss is calculated using a monotonically decreasing function that satisfies Lipshitz continuity. The replacement losses l1 and l2 make the objective function differentiable. Furthermore, due to the monotonicity of σ, by choosing l1 to satisfy the monotonically decreasing function and l1≥0 and l1(0)=1, we can ensure that l1 is the upper bound of the 0-1 loss. Similarly, l2 to satisfy the monotonically decreasing function and l2(0)=0 and l2≤1, we can ensure that l2 is the lower bound of the 0-1 loss. This ensures that the replacement optimization objective (which is to minimize the AUPRC objective loss) is the upper bound of the original objective, ensuring that the relationship between the replacement optimization objective and the original objective is controllable. This achieves the effect of consistency between the optimization objective and the test index. In addition, l1 satisfies the property of a continuous and convex function of Lipshitz, and l2 satisfies Lipshitz continuity, which makes updating the model parameters based on the gradient of the AUPRC objective loss have good convergence. Attached Figure Description

[0052] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0053] Figure 1 This is a flowchart illustrating a training method for an image retrieval model according to an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0055] As mentioned in the background section, existing methods for addressing the non-differentiability of the objective function suffer from several drawbacks. Firstly, the relationship between the objective function after using the substitution loss and the original objective function is difficult to guarantee, and these methods cannot effectively support model algorithm design and theoretical analysis. To address these issues, this invention proposes a scheme to improve the optimization function by starting with AUPRC itself. In summary, this invention optimizes the original AUPRC objective by constructing a target loss after using the substitution loss, ensuring that the AUPRC objective function after the substitution loss serves as an upper bound of the original AUPRC objective. Specifically, this invention uses different substitution losses for different types of samples based on the technical idea of ​​constructing an upper bound of the original AUPRC objective. This ensures that the target loss function after the substitution loss serves as an upper bound of the original target loss function. This guarantees that the target after the substitution loss is consistent with the test metrics and that the target loss function is differentiable during the optimization of image retrieval methods using AUPRC. Ultimately, minimizing the substitution objective effectively optimizes AUPRC to meet the needs of practical scenarios.

[0056] According to one embodiment of the present invention, such as Figure 1As shown, this invention provides a training method for an image retrieval model. The method includes: S1, obtaining a training set and a query image, wherein the training set includes multiple positive sample images of the same class as the query image and multiple negative sample images of different classes from the query image; S2, using the training set and query images obtained in step S1 to train the image retrieval model multiple times until convergence, and updating the image retrieval model parameters using gradient based on the AUPRC target loss during the training process. According to an embodiment of this invention, the AUPRC target loss is:

[0057]

[0058] Where w represents the parameters of the image retrieval model, S represents the set of all samples in the training set, and x + S represents a positive sample image. + x represents all positive sample images in S + The set of positive samples constituted For each positive sample image x in S+, + The corresponding objective function σ(*) values ​​are averaged, where π represents the prior probability obtained beforehand, and h w (x + ) th The similarity threshold for classifying a sample image as a positive sample image is represented by l1, where l1 represents the similarity threshold given h. w (x + ) th The negative sample loss is calculated using a monotonically decreasing convex function that satisfies the Lipshitz continuity condition, where l1≥0 and l1(0)=1. Indicates a given h w (x + ) th The false positive rate corresponding to the training set at time, l2 represents the given h w (x + ) th The positive sample loss is calculated using a monotonically decreasing function that satisfies Lipshitz continuity, and l2(0)=0 and l2≤1. Indicates a given h w (x + The true positive rate corresponding to the training set at that time. It should be noted that the training set and image retrieval model used in this invention are known to those skilled in the art, and will not be described in detail here. The following will further explain the construction, unbiased estimation and experimental verification of the AUPRC target loss.

[0059] I. Construction of AUPRC Target Loss

[0060] The AUPRC target loss provided by this invention is the upper bound of the original AUPRC target loss, which can effectively optimize AUPRC to meet the needs of real-world scenarios. To better understand the advantages of the target loss of this invention, the construction process of the AUPRC target loss in this invention will be explained from two aspects: the introduction of AUPRC target loss in the prior art and the improvement of AUPRC target loss provided by this invention.

[0061] 1.1 AUPRC target loss in existing technology

[0062] As is well known, the image retrieval objective of AUPRC-based image retrieval methods is to learn a scoring function. To maximize the AUPRC of target retrieval, the goal is to make the scores of positive sample images as high as possible compared to negative sample images.

[0063]

[0064] Where w represents the parameters of the image retrieval model, Represents a set of original images. Let h represent the score set. w (x) represents the similarity score between the sample image and the query image, h w (x + ) th This represents the similarity threshold at which a sample image is judged as a positive sample image.

[0065] Formula (1) above is equivalent to:

[0066]

[0067] Furthermore, by estimating the distribution of formula (2) based on the similarity scores of the empirical sample set, the following empirical optimization objective can be derived from formula (2):

[0068]

[0069] Where S represents the set of all samples in the training set (i.e., the empirical sample set), x + S represents a positive sample image. + x represents all positive sample images in S + The set of positive samples constituted Indicates S + Each positive sample image x within + The corresponding objective function σ(*) values ​​are averaged, where π represents the prior probability obtained beforehand, and h w (x + ) th The similarity threshold that determines whether a sample image is a positive sample image, l 0-1 Indicates a given hw (x + ) th The 0-1 loss of the sample image is calculated using the existing 0-1 loss function, when h w When (x) < 0, the 0-1 loss value of the sample image l 0-1 (h w (x))=1, otherwise l 0-1 (h w (x))=0, Indicates a given h w (x + ) th The training set is based on the false positive rate corresponding to the 0-1 loss of the sample images. Indicates a given h w (x + The training set is based on the true positive rate corresponding to the 0-1 loss of the sample images.

[0070] 1.2 Improvements to the AUPRC target loss provided by this invention

[0071] As described above, the existing AUPRC objective loss is limited by the non-differentiability of the 0-1 loss, making it impossible to minimize the objective (i.e., Equation 3) using gradient-based optimization methods. Therefore, it is necessary to select a suitable 0-1 replacement loss to make the objective function differentiable, thereby achieving gradient updates based on the objective function. To ensure that the relationship between the replacement optimization objective and the original objective is controllable, this invention proposes to... and Different substitution losses l1 and l2 are chosen. Due to the monotonicity of σ, by choosing l1 as the upper bound of the 0-1 loss and l2 as the lower bound of the 0-1 loss, it can be ensured that the above substitution optimization objective is the upper bound of the original objective. That is, this ensures that the objective after substitution loss is consistent with the test metric during the optimization of the image retrieval method using AUPRC, thereby ensuring that minimizing the substitution objective can effectively optimize AUPRC and meet the needs of practical scenarios. Therefore, the AUPRC target loss provided by this invention is:

[0072]

[0073] Where w represents the parameters of the image retrieval model, S represents the set of all samples in the training set, and x + S represents a positive sample image. + x represents all positive sample images in S + The set of positive samples constituted Indicates S + Each positive sample image x within + The corresponding objective function σ(*) values ​​are averaged, where π represents the prior probability obtained beforehand, and h w (x + )th The similarity threshold for classifying a sample image as a positive sample image is represented by l1, where l1 represents the similarity threshold given h. w (x + ) th The negative sample loss is calculated using a monotonically decreasing convex function that satisfies the Lipshitz continuity condition, where l1≥0 and l1(0)=1. Indicates a given h w (x + ) th The false positive rate corresponding to the training set at time, l2 represents the given h w (x + ) th The positive sample loss is calculated using a monotonically decreasing function that satisfies Lipshitz continuity, and l2(0)=0 and l2≤1. Indicates a given h w (x + The true positive rate corresponding to the training set at that time. It should be noted that the image retrieval model updated by gradient based on the AUPRC target loss provided by this invention can minimize the AUPRC target loss provided by the invention and ensure consistency with the original AUPRC target (Equation 3). According to an embodiment of the present invention, any one of the following monotonically decreasing convex functions is used to calculate the negative sample loss: smoothing function, hinge loss function, and one-sided Huber function. The negative sample loss calculated based on the monotonically decreasing convex function can ensure that when the similarity score predicted by the image retrieval model for the negative sample image relative to the positive sample image is higher, the corresponding gradient is larger, ensuring that the positive and negative sample image pairs are fully optimized.

[0074] Preferably, the present invention uses a one-sided Huber function to calculate the negative sample loss. According to one embodiment of the present invention, the negative sample loss calculated based on the one-sided Huber function is:

[0075]

[0076]

[0077] Among them, h w (x + ) th h represents the similarity threshold by which a sample image is judged as a positive sample image. w (x - Let represent the similarity score between the negative sample image predicted by the image retrieval model and the query image, and τ1 represent the first hyperparameter, preferably τ1∈(0.01, 0.1). It should be noted that the process of calculating the negative sample loss using other functions in the monotonically decreasing convex function is similar to the calculation process of the negative sample loss based on the one-sided Huber function, and will not be elaborated here.

[0078] According to one embodiment of the present invention, the positive sample loss is:

[0079]

[0080]

[0081] Among them, h w (x + ) th h represents the similarity threshold by which a sample image is judged as a positive sample image. w (x + ) represents the similarity score between the positive sample image predicted by the image retrieval model and the query image, and τ2 represents the second hyperparameter.

[0082] According to one embodiment of the present invention, the positive sample loss is:

[0083]

[0084]

[0085] Among them, h w (x + ) th h represents the similarity threshold by which a sample image is judged as a positive sample image. w (x + ) represents the similarity score between the positive sample image predicted by the image retrieval model and the query image, and τ2 represents the second hyperparameter.

[0086] It should be noted that in the above positive sample loss, when τ2→0, l2→l0-1. As long as an appropriate hyperparameter τ2 is selected, l2 can effectively approximate the 0-1 loss, so that the approximation error of the overall substitution loss is within an acceptable range, which satisfies the conditions required for the convergence guarantee of stochastic optimization. Therefore, the preferred second hyperparameter is: τ2=0.001.

[0087] According to one embodiment of the present invention, during the training process, the training set is sampled multiple times, and a batch sample set corresponding to each round of sampling is obtained. The batch sample set corresponding to each round is used to train the image retrieval model. During each round of training, the model parameters are updated using a gradient method based on the AUPRC target loss of the batch sample set corresponding to that round. The AUPRC target loss of the batch sample set during each round of training is:

[0088]

[0089] Where z represents the batch sample set, z + Represents all positive sample images x in z + The set that is formed Represents the set z + Each positive sample image x within + The corresponding objective function σ(*) values ​​are averaged. Indicates that given h w (x + ) th The false positive rate corresponding to time z Indicates that given h w (x + ) th The true positive rate corresponding to time z, l1′ represents the given h w (x + ) th The negative sample loss corresponding to time z, l2′ represents the loss given h. w (x + ) th The positive sample loss corresponding to time z.

[0090] II. Unbiased estimation

[0091] As is well known, the objective function (minimizing the AUPRC objective loss) in image retrieval methods using AUPRC optimization involves ranking all samples. However, when training with multiple rounds of sampling, the unbiasedness of the random estimate of the gradient obtained from the batch sample set is difficult to guarantee. Simply put, each iteration samples a subset z = z0 of S. + ∪z - , where z + and z - They are positive and negative sample sets, each with n + and n - samples, then with the stochastic gradient corresponding to z When using the gradient estimate of f(w, S) and updating model parameters through methods such as gradient descent, assuming there are M possible values ​​for z, consider the following empirical / expected risk:

[0092]

[0093] Obviously, It is an unbiased estimator of F(w;S), but not necessarily an unbiased estimator of f(w,S). However, when the sample size within the batch is sufficient or the sample score variance is sufficiently small, This is an asymptotically unbiased estimate of f(w, S), and consequently, F(w; S) and F(w) are also asymptotically unbiased estimates of the original risk. Based on the above conclusions, reducing the score variance or increasing the number of positive samples within a batch is beneficial for reducing estimation error. Although introducing a regularization term to reduce variance could be considered, directly using variance regularization would cause high-scoring positive samples to tend to lower their scores, and negative samples to do the opposite, which is inconsistent with the optimization objective. Therefore, this invention proposes a scheme to introduce a positive sample auxiliary vector to increase the number of positive samples in a batch, or on this basis, to introduce a semi-variance regularization term to constrain positive samples with scores below the average and negative samples with scores above the average.

[0094] According to one embodiment of the present invention, when introducing a positive sample auxiliary vector to increase the number of positive samples in a batch, the auxiliary vector is determined in the following manner:

[0095]

[0096] in, Let β represent the auxiliary vector corresponding to the t-th training round. j Let β represent the preset learning rate for the j-th training round. s This represents the preset learning rate corresponding to the s-th training round. This represents the linear interpolation corresponding to the similarity between positive samples and the preset query image in the s-th round of training. The similarity score between the positive sample and the preset query image in the s-th round of training.

[0097] The AUPRC target loss after introducing auxiliary vectors is:

[0098]

[0099] in,

[0100]

[0101]

[0102] in, This represents an auxiliary vector estimated using linear interpolation and exponentially weighted average of the similarity scores between the positive sample images corresponding to the batch sample set and the query image. express The number of interpolations in Indicates the introduction of auxiliary vectors The unbiased true positive rate corresponding to the batch sample set z, l2″ represents the given h w (x + ) th And sometimes introduced Positive sample loss afterward Indicates that given h w(x + ) th The false positive rate corresponding to time z, z - Represents all negative sample images x in z. - The set composed of |z - |for z - The number of negative sample images.

[0103] It should be noted that the calculation process of linear interpolation is well known to those skilled in the art, and will not be elaborated here. It should be further explained that when calculating the loss of a batch sample set, this invention introduces an auxiliary vector, which is equivalent to increasing the number of positive samples within a batch. This can reduce the estimation error when training the image retrieval model using batch sample sets. In other words, iteratively updating the parameters of the image retrieval model based on the losses of multiple batch sample sets can achieve asymptotically unbiased estimation, thereby improving the image retrieval performance.

[0104] According to another embodiment of the present invention, in order to better constrain positive samples with scores below the average and negative samples with scores above the average, and to ensure the unbiasedness of random estimation, the present invention also introduces a semi-variance regularization term loss when calculating the loss of the batch sample set. According to one embodiment of the present invention, the semi-variance regularization term is the loss:

[0105]

[0106] Where λ1 is the third hyperparameter, preferably λ1 = 1, n + h represents the number of positive sample images in the batch sample set z. w (x) represents the similarity score between the sample image in z predicted by the image retrieval model and the query image, μ + Let λ represent the average similarity scores between all positive sample images in z and the query image, where λ2 is the fourth hyperparameter, preferably λ2 = 0.1, and n - z represents the number of negative sample images in z. - Represents all negative sample images x in z. - The set composed of μ - This represents the average of all similarity scores between the negative sample images in z and the query image.

[0107] According to one embodiment of the present invention, when updating model parameters using a gradient based on AUPRC target loss and semivariance regularization loss, the AUPRC target loss f(w, z) and semivariance regularization loss without introducing interpolation vectors are... The sum of these losses constitutes the first total loss, and the model parameters are updated based on this first total loss.

[0108] According to one embodiment of the present invention, when updating model parameters using gradient-based methods based on AUPRC target loss and semivariance regularization term loss, the AUPRC target loss after introducing auxiliary vectors will be... and semivariance regularization term loss The sum of these losses constitutes the second total loss, and the model parameters are updated based on this second total loss.

[0109] The model trained using the method of this invention can be directly used for image retrieval.

[0110] As can be seen from the above embodiments, the present invention optimizes the AUPRC objective function by introducing a negative sample loss l1 that satisfies the upper bound of the 0-1 loss and a positive sample loss l2 that satisfies the lower bound of the 0-1 loss, making the optimized objective function differentiable. This ensures the controllability of the optimized objective function and the original objective function, achieving the effect of consistency between the optimization objective and the test index. In addition, the present invention also introduces an auxiliary vector for positive sample image estimation and a semivariance regularization loss in random estimation to achieve the generalizability and asymptotic unbiasedness of the substitute objective.

[0111] III. Experimental Verification

[0112] To better illustrate the technical effects of the present invention, the following experiments were conducted for verification.

[0113] First, in this experiment, the image retrieval model uses ResNet-50 residual neural network as the feature extractor, and the similarity between the query image and the sample image is calculated using cosine similarity. The input to the feature extractor is a 224×224 color image of the sample image, and the output of the feature extractor is a 512-dimensional embedding.

[0114] Furthermore, based on the above-mentioned image retrieval model configuration, this experiment uses three training sets to train the image retrieval model. For each training set, eight existing losses are used as a control experiment against the AUPRC target loss provided in this invention. A test set is then constructed using the datasets corresponding to the trained image retrieval model to evaluate it. The three training sets are the Stanford Online Product-SOP product retrieval dataset, the iNaturalist long-tail species retrieval dataset, and the PKU VehicleID dataset. The eight existing losses include four pairwise loss-based losses (Contrastive loss, Triplet loss, Multi-similarity (MS) loss, and XBM) and losses based on AUPRC-optimized retrieval methods (SmoothAP, DIR, FastAP, and BlackBox). The image retrieval model is evaluated using the AUPRC metric and the ratio of the number of relevant results retrieved from the top K results to the total number of relevant results in the database. These eight existing losses serve as the control experiment results for this invention. Detailed results are shown in Table 1.

[0115] Table 1

[0116]

[0117]

[0118] In Table 1, Ours represents the AUPRC target loss provided by this invention, mAUPRC represents the AUPRC metric used to evaluate the image retrieval model, and R@1, R@4, R@5, and R@10 represent the recall rates of the number of relevant results retrieved in the first 1, 4, 5, and 10 results, respectively, and the total number of relevant results in the database. The numbers are highlighted and underlined (e.g., the experimental results in Table 1). The result represents the optimal experimental outcome after training the image retrieval model based on loss, using only the bolded numbers (e.g., the experimental results in Table 1). The result represents the suboptimal experimental result after training the image retrieval model based on the loss. Among the experimental results, only the result where the R@1 metric of the AUPRC target loss provided in this invention is close to optimal in image retrieval of the vehicle re-identification dataset (PKU VehicleID) is achieved (i.e.,...). near In this experiment, the experimental results for the remaining datasets were optimal. As can be seen from Table 1, the method of training the image retrieval model using the AUPRC target loss provided by this invention significantly outperforms the best existing methods on multiple datasets and metrics, effectively verifying the effectiveness of this invention.

[0119] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0120] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0121] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0122] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A training method for an image retrieval model, characterized in that, The method includes: S1. Obtain a training set and a query image. The training set includes multiple positive sample images of the same class as the query image and multiple negative sample images of different classes from the query image. S2. Using the training set and query image obtained in step S1, the image retrieval model is trained multiple times until convergence. During the training process, the parameters of the image retrieval model are updated using gradient-based methods based on the AUPRC target loss, wherein the AUPRC target loss is: in, The parameters represent the image retrieval model. Represents the set of all samples in the training set. Represents a positive sample image. express All positive sample images The set of positive samples constituted Indicates to Each positive sample image within Corresponding objective function Take the average value. This represents the prior probability obtained in advance. This represents the similarity threshold by which a sample image is classified as a positive sample image. Indicates a given The negative sample loss is calculated using a monotonically decreasing convex function that satisfies the Lipshitz continuity condition, and as well as , Indicates a given The false positive rate corresponding to the training set at that time. Indicates a given The positive sample loss is calculated using a monotonically decreasing function that satisfies Lipshitz continuity, and as well as , Indicates a given The true positive rate corresponding to the training set; where, any one of the following monotonically decreasing functions is used to calculate the negative sample loss: smoothing function, hinged loss function, one-sided Huber function, where the negative sample loss calculated based on the one-sided Huber function is: in, This represents the similarity threshold by which a sample image is classified as a positive sample image. This represents the similarity score between the negative sample image predicted by the image retrieval model and the query image. Indicates the first hyperparameter; The positive sample loss is: in, This represents the similarity threshold by which a sample image is classified as a positive sample image. This represents the similarity score between the positive sample image predicted by the image retrieval model and the query image. This represents the second hyperparameter.

2. The method according to claim 1, characterized in that, The positive sample loss is: in, This represents the similarity threshold by which a sample image is classified as a positive sample image. This represents the similarity score between the positive sample image predicted by the image retrieval model and the query image. This represents the second hyperparameter.

3. The method according to claim 1 or 2, characterized in that, In step S2, the training set obtained in step S1 is sampled multiple times. Each round of sampling yields a corresponding batch sample set, which is then used to train the image retrieval model. During each training round, the image retrieval model parameters are updated using a gradient based on the AUPRC target loss of the corresponding batch sample set. The AUPRC target loss for the batch sample set during each training round is: in, Represents the batch sample set. express All positive sample images The set that is formed Represents a set Each positive sample image within Corresponding objective function Take the average value. Indicates that in a given hour The corresponding false positive rate, Indicates that in a given hour The corresponding true yang rate, Indicates a given hour The corresponding negative sample loss, Indicates a given hour The corresponding positive sample loss.

4. The method according to claim 3, characterized in that, When calculating the loss for the batch sample set, an auxiliary vector is introduced and the AUPRC target loss is calculated as follows: in, in, This represents an auxiliary vector estimated using linear interpolation and exponentially weighted average of the similarity scores between the positive sample images corresponding to the batch sample set and the query image. The number of interpolations in Indicates the introduction of auxiliary vectors Time batch sample set The corresponding unbiased true yang rate, Indicates a given When introduced The corresponding positive sample loss.

5. The method according to claim 4, characterized in that, The auxiliary vector is determined in the following way: in, Indicates the first The auxiliary vectors corresponding to each round of training. Indicates the first The preset learning rate corresponding to each round of training. Indicates the first The preset learning rate corresponding to each round of training. Indicates the first Linear interpolation corresponding to the similarity between positive samples and preset query images during training rounds. For the first Similarity scores between positive samples and preset query images during training rounds.

6. The method according to claim 5, characterized in that, In step S2, a semivariance regularization loss is also introduced, and the model parameters are updated using gradient descent based on the AUPRC objective loss and the semivariance regularization loss, where the semivariance regularization term is the loss term. in, This is the third hyperparameter. Represents the batch sample set The number of positive sample images in the data. Indicates the prediction of the image retrieval model Similarity score between the sample image and the query image. express The average of all similarity scores between the positive sample image and the query image. It is the fourth hyperparameter. express The number of negative sample images in the data. express All negative sample images The set that is formed express The average of all similarity scores between the negative sample images and the query image.

7. The method according to claim 6, characterized in that, （0.01，0.1）， ， ， 。 8. An image retrieval method, characterized in that, The image retrieval method includes: T1. Obtain the query image; T2. Using the image retrieval model obtained by any one of the methods described in claims 1-7, the retrieval results of the query image are obtained in the image database.

9. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the electronic device to perform the steps of the method as described in any one of claims 1 to 7.

Images