A Deep Metric Learning Adversarial Defense Method Based on Adaptive Weighting

Abstract

The present invention discloses a deep metric learning adversarial defense method based on adaptive weighting, which designs an adaptive weighted loss L for image positive sample pairs and image negative sample pairs, wherein the weight value W of the image positive sample pair is p It is an increasing function in the range [0,1], and the weight value W of the image negative sample pair N A decreasing function in the range [0, 1] allows for adaptively assigning appropriate weights to different image sample pairs based on their distance scores, thus adaptively manipulating the attack strength of image sample pairs at different training stages. This weakens the attack strength of adversarial examples in early training stages and strengthens it in later training stages. Compared to existing methods that manually formulate attack strategies, this method significantly improves the efficiency of adversarial training and the adversarial robustness of image deep metric learning models.

Description

Technical Field

[0001] The present invention belongs to the technical field of image deep metric learning models, and more specifically, relates to a deep metric learning adversarial defense method based on adaptive weighting. Background Art

[0002] Deep metric learning adversarial defense methods aim to improve the vulnerability of existing metric learning models to attacks. Ranking systems based on deep metric learning are applied to a variety of practical tasks, such as fine-grained image retrieval, vehicle retrieval, and person re-identification. Existing metric learning models are vulnerable to attacks when human-imperceptible perturbations are injected into the query image, causing the ranking system to produce an unreasonable ranking. Deep metric learning adversarial defense methods ensure the fairness and security of the system by improving the adversarial robustness of the ranking model.

[0003] Existing adversarial defense methods for deep models fall into four main categories: input transformation, randomization, model ensembles, and adversarial training. Input transformation-based methods use image preprocessing operations such as image compression and denoising to enhance images before they are used for model training. Randomization-based methods add random noise to the original image or model parameters to improve the model's robustness to noise, including adversarial perturbations. Model ensemble methods further enhance model robustness by integrating multiple weak defenses in the output layer. Adversarial training methods, currently the mainstream defense method for deep models, improve the model's adversarial robustness through min-max adversarial training.

[0004] Minimum-maximum adversarial training consists of an adversarial image maximization problem and a model minimization problem. The adversarial image maximization problem focuses on synthesizing adversarial images of the input image, while the model minimization problem is dedicated to using adversarial images to update model parameters. Existing methods for adversarial image generation propose methods such as one-step gradient descent and projected gradient descent to improve the attack performance of adversarial images on deep models. Regarding the problem of using adversarial samples to update model parameters, based on basic adversarial training, the effectiveness and efficiency of adversarial training are improved by studying issues such as searching for different attack strategies at different training stages, how to adjust the training process based on the characteristics of the adversarial loss during training, and how to reduce the size of adversarial images used for adversarial training.

[0005] The performance of min-max adversarial training often depends on the strength of the generated adversarial images. While existing adversarial defense methods have made significant progress in improving robustness, most generate adversarial images by fixing the parameters of PGD (projected gradient descent) or its variants, making it difficult to control the strength of the generated adversarial images. Recent research has shown that using adversarial samples (images) of varying strengths at different training stages can improve the adversarial robustness of a model, namely, using weak adversarial samples (images) in early training stages and strong adversarial images in later training stages. However, due to the specific nature of min-max adversarial training, adversarial samples (images) generated in the early stages have excessive strength. As adversarial training progresses, the generated adversarial images become less effective against the trained model. Most methods manually craft attack strategies to control the strength of adversarial images, which requires expert knowledge and has limited robustness improvements. Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a deep metric learning adversarial defense method based on adaptive weighting, which uses simple adversarial images in the early training stage and more difficult adversarial images in the later training stage to improve the adversarial robustness of the image deep metric learning model.

[0007] To achieve the above-mentioned object of the invention, the present invention provides an adaptive weighted deep metric learning adversarial defense method, which is characterized by comprising the following steps:

[0008] (1) Construct adaptive weighted loss L:

[0009]

[0010] Where N is the number of distance scores used in training, and W p Represents the weight value of the positive sample pair of the image, and W n Represents the weight value of the image negative sample pair, is the distance score of the positive sample pair of the image, is the distance score of the image negative sample pair, λ is a constant, and the plus sign in the lower right corner indicates that the adaptive weighted loss L takes a positive value;

[0011] W p and W n are all defined as functions of distance fractions:

[0012]

[0013]

[0014] Where F(·) is defined as the weighting function, α and β represent the parameter set of the weighting function;

[0015] Among them, the weight value W of the positive sample pair of the image p It is an increasing function in the range [0, 1], and the weight value W of the image negative sample pair n It is a decreasing function in the range [0, 1];

[0016] (2) Based on minimum-maximum adversarial training, generate adversarial images and obtain r intra-class distance scores, that is, the set of distance scores of image positive samples. And s inter-class distance scores, that is, the set of image negative sample distance scores Then, N distance scores are selected from the two sets respectively, and the image depth metric learning model is learned based on the adaptive weighted loss L.

[0017] The object of the present invention is achieved in this way.

[0018] The present invention designs an adaptive weighted loss L for the image positive sample pair and the image negative sample pair based on the adaptive weighted deep metric learning adversarial defense method, where the weight value W of the image positive sample pair is p It is an increasing function in the range [0, 1], and the weight value W of the image negative sample pair N The proposed method uses a decreasing function in the range [0, 1]. This allows for adaptively assigning appropriate weights to different image sample pairs based on their distance scores, thus adaptively manipulating the attack strength of image sample pairs at different training stages. This approach weakens the attack strength of adversarial samples in the early training stages and strengthens it in the later stages. Compared to existing methods that manually formulate attack strategies, this method effectively improves the training efficiency of adversarial training and the adversarial robustness of image deep metric learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the present invention's deep metric learning adversarial defense method based on adaptive weighting;

[0020] Figure 2 Schematic diagram of the principle of the adaptively weighted deep metric learning adversarial defense method of the present invention, wherein (a) is before weight adjustment in the early stage of training, (b) is after weight adjustment in the early stage of training, (c) is before weight adjustment in the late stage of training, and (d) is after weight adjustment in the late stage of training;

[0021] Figure 3 The present invention is a graph of weights and distance scores between positive and negative samples;

[0022] Figure 4 This is a curve diagram of the adversarial sample strength of the present invention and the existing method HM. DETAILED DESCRIPTION

[0023] The following describes the specific embodiments of the present invention in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, such descriptions will be omitted here.

[0024] During adversarial training, the strength of the generated adversarial samples has a key impact on the robustness of the model. Recent studies have shown that using overly strong adversarial samples (images) for adversarial training in the early training stage can cause the model to crash. The results of these works show that using simple adversarial samples (images) in the early training stage and more difficult adversarial samples (images) in the later training stage can improve the robustness of the model. However, due to the particularity of minimum-maximum adversarial training, the adversarial samples (images) generated in the early stage have too large an attack strength. As the adversarial training proceeds, the generated adversarial samples (images) are weaker in attack against the trained model, which creates a contradiction. The present invention proposes a deep metric learning adversarial defense method based on adaptive weighting, which adaptively controls the strength of adversarial sample pairs based on distance scores. In this way, in the early training stage, the present invention assigns appropriate weight values ​​to adversarial sample pairs to weaken their attack strength. Vice versa, their attack power is enhanced in the later stages of training.

[0025] Figure 1 This is a flow chart of the present invention's deep metric learning adversarial defense method based on adaptive weighting.

[0026] Specifically, if Figure 1 As shown, the adaptive weighted deep metric learning adversarial defense method of the present invention includes the following steps:

[0027] Step S1: Construct an adaptive weighted loss L:

[0028]

[0029] Where N is the number of distance scores used in training, and W p Represents the weight value of the positive sample pair of the image, and W n Represents the weight value of the image negative sample pair, is the distance score of the positive sample pair of the image, is the distance score of the image negative sample pair, λ is a constant, and the plus sign in the lower right corner indicates that the adaptive weighted loss L takes a positive value;

[0030] W p and W n are all defined as functions of distance fractions:

[0031]

[0032]

[0033] Where F(·) is defined as the weighting function, α and β represent the parameter set of the weighting function;

[0034] Among them, the weight value W of the positive sample pair of the image p It is an increasing function in the range [0, 1], and the weight value W of the image negative sample pair n It is a decreasing function in the range [0, 1].

[0035] Step S2: Learning the image depth metric learning model based on the adaptive weighted loss L

[0036] Based on minimum-maximum adversarial training, adversarial images are generated and r intra-class distance scores are obtained, that is, a set of distance scores of image positive sample pairs. And s inter-class distance scores, that is, the set of image negative sample distance scores Then, N distance scores are selected from the two sets respectively, and the image depth metric learning model is learned based on the adaptive weighted loss L.

[0037] Figure 2 It is a schematic diagram of the principle of the adaptive weighted deep metric learning adversarial defense method of the present invention.

[0038] The present invention uses different weighting functions for positive and negative image sample pairs. Due to the particularity of minimum-maximum adversarial training, the image adversarial samples generated in the early stage have too strong an attack strength. As the adversarial training progresses, the generated image adversarial samples become less effective against the trained model. To this end, the present invention sets appropriate parameters for α and β so that the weight value W of the positive image sample pair is p It is an increasing function in the range [0, 1], and the weight value W of the image negative sample pair N It is a decreasing function in the range of [0, 1] to weaken the attack strength of the adversarial samples generated in the early stage and enhance the attack strength in the later stage. Figure 2 As shown, in the early stage of training, the image x a_adv Its positive sample x p_adv The distance score is larger than that of its negative sample x p_adv The distance score is small and the attack strength is too large, such as Figure 2 (a) As shown. After weight adjustment according to the present invention, image x a_adv Its positive sample x p_adv The distance score decreases with its negative sample x p_adv The distance score of is increased to weaken its attack strength, such as Figure 2 (b) As shown. In the later stage of training, the image xa_adv Its positive sample x p_adv The distance score is small, and its negative sample x p_adv The distance score is large, and the attack strength is weak, such as Figure 2 (c) As shown. After weight adjustment according to the present invention, the image x a_adv Its positive sample x p_adv The distance score increases with its negative sample x p_adv The distance score of is reduced to enhance its attack strength, such as Figure 2 (d) shown.

[0039] Figure 3 The present invention is a curve graph of weights and distance scores of positive and negative samples.

[0040] In this embodiment, if Figure 3 As shown, the function of the distance score controls the weight of the strength of the adversarial sample based on the sample distance score. The strength of the adversarial sample can be indirectly measured by the sample distance score. The smaller the distance score of the negative sample pair, the greater the strength, and the smaller the distance score of the positive sample pair, the greater the strength. Since the distance score is expressed as an increasing function for negative samples and a decreasing function for positive samples as training progresses (the strength of the adversarial sample decreases), and the value range is [0, 1], in order to achieve the weakening of sample strength in the early stage and the strengthening in the later stage, the function target of the distance score is: within the value range of [0, 1], the distance score function of the negative sample pair is a decreasing function, and the distance score function of the positive sample pair is an increasing function. Among them, Figure 3 Three decreasing function curves and three increasing function curves are given respectively, and the specific curves are determined according to the specific implementation situation.

[0041] Based on the adaptive weighted loss proposed above for constructing the relationship between weight values ​​and distance scores of image sample pairs, in this embodiment, a polynomial loss function is designed for minimum-maximum adversarial training. The weights of the positive and negative image sample pairs are defined as:

[0042]

[0043]

[0044] Among them, α k and β h are hyperparameters, k=0,1,...,K, h=0,1,...,H, α={α0,α1,...,α K}, β={β0, β1,...,β H}.

[0045] Substituting the weights into the adaptive weighted loss L proposed in the present invention, we get:

[0046]

[0047] Where [x] + =max{x, 0} represents the number of tuples, K and H are the highest orders of the polynomial function. Benefiting from the ability of the polynomial function to approximate any continuous function, the weighting function used in the present invention has great flexibility in form.

[0048] Loss gradient analysis

[0049] To further demonstrate the superiority of the proposed method, we calculated the polynomial loss gradient for positive and negative pairs respectively. It can be expressed as:

[0050]

[0051] and

[0052]

[0053] where α k and β h is a hyperparameter.

[0054] It can be seen from the above two equations that the gradient values ​​of the positive and negative pairs are related to their distance scores, so the polynomial weighted loss used in this embodiment can make the adversarial training optimization process more flexible.

[0055] Theoretical analysis

[0056] Different from traditional weighting methods that use a monotonic weighting function (monotonically increasing or monotonically decreasing) throughout the training process, our proposed polynomial weighting function allows for a more flexible weighting scheme during training.

[0057] Experimental verification

[0058] To verify the effectiveness of the proposed method, we tested the deep metric model's adversarial defense on the CUB and CARS datasets using different maximum attack step sizes. We used recall, mean average precision (mAP), and normalized mutual information (NMI) to evaluate the defense method proposed in this patent. Recall refers to the average number of samples in the top k nearest neighbors of a given query sample that have at least one sample of the same category as the query sample. Mean average precision (mAP) jointly considers overall ranking information and precision by calculating the average precision (AP) of all queries. Normalized mutual information measures the clustering quality and the degree of similarity by combining the mutual information between the clustering results and the labels, as well as the entropy information of the clustering results and the labels. To further evaluate the model's adversarial robustness, this patent also uses the empirical robustness score (ERS), a standardized score for deep metric model adversarial attacks, to test the model. The empirical robustness score evaluates the model's defense performance by measuring the defense success rate against a series of simple white-box attacks.

[0059] As shown in Tables 1 and 2, the method proposed in the present invention achieves the best performance on both datasets. This method not only improves the robustness of the model based on the adversarial training process, but also ensures the accuracy of the original input image. This phenomenon shows that our method (the present invention) achieves a balance between accuracy and robustness. Compared with the previous optimal method, our method improves R@1: 2.0% / 1.2%, R@2: 1.6% / 0.7%, NMI: 1.3% / 0.8%, and ERS: 4.4% / 3.7% on the two datasets when the maximum attack step size is 32. The method proposed in the present invention achieves a balance between the empirical robustness score and the recall score under different maximum attack step sizes. Under some settings, the recall score of the present invention is slightly lower than the existing best method, but the empirical robustness score result is much higher, which shows that the proposed model is efficient and converges quickly.

[0060]

[0061] Table 1

[0062]

[0063] Table 2

[0064] Figure 4 This is a curve diagram of the adversarial sample strength of the present invention and the existing method HM.

[0065] Figure 4 The effectiveness of our method is indirectly demonstrated by the strength of adversarial examples, as represented by the distance between the original and adversarial examples. For negative pairs, the smaller the distance between the original and adversarial examples (the greater the similarity), the stronger the adversarial example. The figure shows that compared to the existing HM method, the proposed method has a larger distance (lower adversarial example difficulty) in the early stages of training and a smaller distance (higher adversarial example difficulty) in the later stages of training. Overall, the HM distance curve is an increasing function (continuously decreasing adversarial example difficulty) for most stages of training, while the proposed method shows a decreasing function with a smaller slope (slower adversarial example difficulty). These observations indicate that the experimental results are consistent with expectations, optimizing the strength of adversarial examples used at different stages of training.

[0066] Although the above describes the illustrative specific embodiments of the present invention to facilitate understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concepts of the present invention are protected.

Claims

1. A deep metric learning adversarial defense method based on adaptive weighting, characterized in that: The following steps are involved: (1) Construct adaptive weighted loss L: Where N is the number of distance scores used in training, and W p Represents the weight value of the positive sample pair of the image, and W n Represents the weight value of the image negative sample pair, is the distance score of the positive sample pair of the image, is the distance score of the image negative sample pair, λ is a constant, and the plus sign in the lower right corner indicates that the adaptive weighted loss L takes a positive value; W p and W n are all defined as functions of distance fractions: Where F(·) is defined as the weighting function, α and β represent the parameter set of the weighting function; Among them, the weight value W of the positive sample pair of the image p It is an increasing function in the range [0, 1], and the weight value W of the image negative sample pair n It is a decreasing function in the range [0, 1]; (2) Based on minimum-maximum adversarial training, generate adversarial images and obtain r intra-class distance scores, that is, the set of distance scores of image positive samples. And s inter-class distance scores, that is, the set of image negative sample distance scores Then, N distance scores are selected from the two sets respectively, and the image depth metric learning model is learned based on the adaptive weighted loss L.

2. The method for adversarial defense based on adaptive weighted deep metric learning according to claim 1, characterized in that: The weights of the positive and negative sample pairs of the image are: where α k and β h are hyperparameters, k = 0, 1, ..., K, h = 0, 1, ..., H, α = {α0, α1, ..., α K}, β = {β0, β1, ..., β H}.

Images