Adapter-based pedestrian re-identification robustness improvement method

Abstract

The invention discloses a pedestrian re-recognition robustness improvement method based on an Adapter, and the method comprises the steps: employing a Transform as an image encoder of a backbone network, introducing an additional adversarial token ([ADV] token), and enabling the adversarial token and a low-rank matrix to serve as adapters for training. The method specifically comprises the following steps: in a training stage, fixing parameters of an image encoder, inputting an image into the encoder, and generating an adversarial sample by using two-step PGD iteration in combination with adversarial cross entropy loss and adversarial triple loss; splicing an original image with a classification token [CLS], splicing an adversarial sample with an adversarial token [ADV], superposing a learnable low-rank matrix, and jointly inputting into an image encoder; an image encoder and an adapter are trained by jointly calculating loss functions of an original image and an adversarial sample. In a test stage, image features are extracted for a pedestrian re-identification task. The method is simple to implement, the robustness and retrieval accuracy of the model to the attack image are remarkably improved, and a more reliable technical means is provided for a pedestrian re-recognition system.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and digital image processing, and in particular to a method for improving the robustness of person re-identification using an adapter-based Transformer backbone network. Background Art

[0002] Person re-identification technology aims to accurately retrieve images of people with the same identity as a given query image from a large library of images taken by different cameras. Recent studies have shown that the Transformer network exhibits significant performance advantages in person re-identification compared to the traditional convolutional neural network (CNN) framework. By introducing stacked self-attention layers, the Transformer network can deeply understand and analyze the relationships between all tags in the image and update them in a dynamic manner. As a result, the network has a wide receptive field and can capture global and discriminative features, thereby improving the accuracy of person re-identification. In addition, the Transformer network is generally considered to have a stronger ability to resist adversarial attacks and is more robust and reliable than CNN.

[0003] Although Transformer networks have demonstrated significant performance improvements over traditional CNN frameworks in person re-identification tasks, effectively capturing global and discriminative features through stacked self-attention layers, recent experiments have shown that Transformer network performance metrics significantly degrade when faced with attacks specifically targeting person re-identification. Furthermore, Transformer networks have not demonstrated a significant advantage in performance stability over CNN-based models, contrary to previous expectations.

[0004] Therefore, in order to ensure the reliability and security of the Transformer network in the person re-identification task, people need to adopt an effective defense strategy to resist malicious attacks. Summary of the Invention

[0005] To address the vulnerability of Transformer networks in person re-identification (ReID) tasks, this paper proposes an adversarial training strategy to build a robust model. This strategy aims to significantly improve the robustness of the Transformer model while ensuring that the performance impact on clean images remains within a range of slight degradation.

[0006] The object of the present invention is achieved like this:

[0007] An adapter-based method for improving the robustness of person re-identification (PRI) is proposed. An adapter is constructed by introducing adversarial tokens and a low-rank matrix into the image encoder of the Transformer structure to improve robustness under attack conditions. The method includes the following steps:

[0008] Step 1: During the training phase, the parameters of the image encoder are fixed and the image is fed into the image encoder.

[0009] Step 2: Calculate the inverse cross entropy loss between the image and the identity and the adversarial ternary loss function between images , and use the two-iteration PGD attack method to generate adversarial samples. The corresponding formula is as follows;

[0010]

[0011]

[0012] in is the total number of identities of all characters in the training set, is the identity index, The model predicts The probability of the class, Is sample i belongs to The true label of the class, is the image feature output by the image encoder for the i-th image, It is to calculate the Euclidean distance between two image features. is the farthest negative sample in the current batch that is inconsistent with the identity of image i, is the most recent positive sample in the current batch that has the same identity as image i, is a fixed threshold;

[0013] Step 3: Input the original image and the adversarial sample into the image encoder at the same time and encode them into corresponding image token sequences respectively;

[0014] Step 4: Concatenate the original image token with the classification token [CLS] in the Transformer backbone network; at the same time, concatenate the adversarial sample token with the newly added adversarial token [ADV], and introduce a learnable low-rank matrix to add to the adversarial sample token;

[0015] Step 5: Calculate the loss of the original image and the adversarial example separately, and jointly train the image encoder and the newly added adapter;

[0016] The loss of the original image and the adversarial sample is the identity loss function and triplet loss function , calculated as follows:

[0017]

[0018]

[0019] in The probability of the model predicting the kth category, The one-hot encoding of the true label of the k-th id, and Represents the distance from positive samples and negative samples, yes Set thresholds;

[0020] Step 6: During the testing phase, the test set images are fed into the trained image encoder to obtain corresponding image features for person re-identification. For each image in the query value of the test set, the most similar image taken by other cameras is found in the candidate image set gallery. The mAP and Rank-1 metrics are calculated. The higher the metrics, the better the results.

[0021] Furthermore, the backbone network of the image encoder is ViT-B / 16 of the Transformer network, and the newly added adapter is Adapter. In the Adapter, the newly added adapter is a structure composed of an adversarial token and a low-rank matrix; the adversarial token is part of the adapter and is used to replace the classification token [CLS] token in the backbone network when processing adversarial samples. Its position and dimension are consistent with the [CLS] token and it is spliced ​​at the front end of the image token sequence; the low-rank matrix is ​​another part of the adapter and is introduced at the input stage of the image encoder. It has the same dimension as the image token and the same position as the position encoding layer and is added to the image token of the adversarial sample.

[0022] Furthermore, the newly added adversarial token [ADV] token and low-rank matrix are combined with the adversarial sample token as adapters; the classification token [CLS] token of the Transformer backbone network is combined with the original image token; the newly added adapter Adapter and the classification token of the original image are independent of each other and do not share information; thereby achieving structural distinction between the adversarial sample and the original image in the input representation, thereby improving the discriminative ability and robustness of the model under attack conditions.

[0023] Furthermore, the test set is subdivided into two categories: an image set that has not been attacked and an image set that has been attacked; the attacked test set is obtained by processing the images in the original test set using a variety of pedestrian re-identification attack methods, including PGD, UAP, Meta Attack, etc.; during the testing phase, one of the data sets is selected for inference to evaluate the performance of the model under original conditions and under attacked conditions respectively. The evaluation indicators include mean average accuracy (mAP) and Rank-1 recognition rate, and the higher the value, the stronger the model robustness.

[0024] This method uses a Transformer-based image encoder and newly added [ADV] tokens and low-rank matrices for training: The image encoder parameters are fixed and the image is fed into the image encoder. Adversarial cross-entropy loss and adversarial ternary loss are calculated to generate attack samples. The original image is combined with the original classification tokens, and the adversarial sample is combined with the added adversarial tokens, mapped into a learnable low-rank matrix, and fed into the image encoder. Losses for the image and identity are calculated, and the image encoder and newly added parameters are trained. The image encoder then generates features for person re-identification. This method incorporates adversarial learning into a Transformer-based ReID model and introduces independent [ADV] tokens and low-rank matrices to handle adversarial samples. This simple method improves the model's robustness against previously unretrievable query images or gallery images that have been subjected to special attack processing, thereby enhancing the model's stability and accuracy against specific attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the process of the present invention;

[0026] Figure 2 This is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to more clearly illustrate the technical means, technical improvements and beneficial effects of the present invention, the present invention is described in detail below with reference to the accompanying drawings.

[0028] Example 1

[0029] See Figure 1-2 , an adapter-based method for improving the robustness of pedestrian re-identification is to add an additional set of tokens and low-rank matrices to the Transformer image encoder as adapter training, and improve the robustness of the image encoder under attacks in the pedestrian re-identification task. Specifically, the following steps are included:

[0030] S1: For the Market1501 dataset with 750 identities in the training set and 751 identities in the test set, during the training phase, the parameters of the image encoder are fixed and the image is fed into the image encoder.

[0031] S2: Calculate the reverse cross entropy loss between the image and the identity and the adversarial ternary loss function between images , and use the two-iteration PGD attack method to generate adversarial samples. The corresponding formula is as follows;

[0032]

[0033]

[0034]

[0035] in is the total number of identities of all characters in the training set, is the identity index, The model predicts The probability of the class, Is sample i belongs to The true label of the class is 1 when k=y and 0 otherwise. is the image feature output by the image encoder for the i-th image, To calculate the Euclidean distance between two image features, is the farthest negative sample in the current batch that is inconsistent with the identity of image i, is the most recent positive sample in the current batch that has the same identity as image i, is a fixed threshold value, set to 0.3. is the maximum infinity norm value of the perturbation, set to 0.3125, t is the current iteration number, sgn is the sign function. x(t) represents the adversarial image of the t-th iteration, To generate the maximum infinity norm value of the perturbation, set it to 0.3125. Indicates finding the gradient of the image input. The updated image is re-projected back to the original image x as the center and The formula indicates that the image is perturbed using the gradient direction of the joint adversarial loss, and the final adversarial image is obtained through two iterations.

[0036] Step 3: The original image and the adversarial sample are fed into the image encoder together, and after encoding, they are converted into the corresponding token sequence.

[0037] Step 4: Concatenate the image token obtained by mapping the original image with the classification token [CLS] token. At the same time, add an [ADV] token of the same size as the [CLS] token and concatenate it with the image token mapped by the adversarial sample. A learnable low-rank matrix is ​​introduced and added to the image token. The matrix size is the same as the image token. The processed token sequence is then input into the image encoder.

[0038] Step 5: Calculate the loss of the adversarial sample and the original image respectively, and train the image encoder and the added adapter for training;

[0039] Step 6: During the testing phase, the test set images are fed into the trained image encoder to obtain the corresponding image features for person re-identification: For each image in the test set query, the most similar image from another camera is found in the gallery, and the mAP and Rank-1 metrics are calculated. For the transformer model without this method, the mAP is 87.4 and the Rank-1 is 94.2 on the unattacked test set; for the query image after the meta-attack, the mAP is 17.4 and the Rank-1 is 18.0. For the transformer model trained using this method, the mAP is 86.9 and the Rank-1 is 94.5 on the unattacked test set; for the query image after the meta-attack, the mAP is 70.1 and the Rank-1 is 82.5.

[0040] The purpose of this invention is to propose an adversarial training strategy to build a robust model. This strategy aims to significantly improve the robustness of the Transformer model while ensuring that the performance impact on clean images remains within a slightly degraded range. The above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for improving the robustness of person re-identification based on an adapter, characterized by: The method comprises the following steps: Step 1: During the training phase, the parameters of the image encoder are fixed and the image is fed into the image encoder. Step 2: Calculate the inverse cross entropy loss between the image and the identity and the adversarial ternary loss function between images , and use the two-iteration PGD attack method to generate adversarial samples. The corresponding formula is as follows; ； ； in is the total number of identities of all characters in the training set, is the identity index, The model predicts The probability of the class, Is sample i belongs to The true label of the class, is the image feature output by the image encoder for the i-th image, It is to calculate the Euclidean distance between two image features. is the farthest negative sample in the current batch that is inconsistent with the identity of image i, is the most recent positive sample in the current batch that has the same identity as image i, is a fixed threshold; Step 3: Input the original image and the adversarial sample into the image encoder at the same time and encode them into corresponding image token sequences respectively; Step 4: Concatenate the original image token with the classification token [CLS] in the Transformer backbone network; at the same time, concatenate the adversarial sample token with the newly added adversarial token [ADV], and introduce a learnable low-rank matrix to add to the adversarial sample token; Step 5: Calculate the loss of the original image and the adversarial example separately, and jointly train the image encoder and the newly added adapter; The loss of the original image and the adversarial sample is the identity loss function and triplet loss function , calculated as follows: ； ； in The probability of the model predicting the kth category, The one-hot encoding of the true label of the k-th id, and Represents the distance from positive samples and negative samples, yes Set thresholds; Step 6: During the testing phase, the test set images are fed into the trained image encoder to obtain corresponding image features for person re-identification. For each image in the query value of the test set, the most similar image taken by other cameras is found in the candidate image set gallery. The mAP and Rank-1 metrics are calculated. The higher the metrics, the better the results.

2. The method for improving robustness of person re-identification according to claim 1, characterized in that: The backbone network of the image encoder is the ViT-B / 16 of the Transformer network, and the newly added adapter is an adapter. In the adapter, the newly added adapter is a structure composed of an adversarial token and a low-rank matrix. The adversarial token, as part of the adapter, is used to replace the classification token [CLS] token in the backbone network when processing adversarial samples. Its position and dimension are consistent with the [CLS] token and it is spliced ​​at the front end of the image token sequence. The low-rank matrix is ​​introduced as another part of the adapter at the input stage of the image encoder, has the same dimension as the image token, and is in the same position as the position encoding layer, and is added to the image token of the adversarial sample.

3. The method for improving robustness of person re-identification according to claim 1, characterized in that: The newly added adversarial token [ADV] token and low-rank matrix are combined with the adversarial sample token as adapters; the classification token [CLS] token of the Transformer backbone network is combined with the original image token; the newly added adapter Adapter and the classification token of the original image are independent of each other and do not share information; thereby achieving structural distinction between the adversarial sample and the original image in input representation.

4. The method for improving robustness of person re-identification according to claim 1, characterized in that: The test set is divided into two categories: an image set that has not been attacked and an image set that has been attacked. The attacked test set is obtained by processing the images in the original test set using a variety of person re-identification attack methods, including PGD, UAP, Meta Attack, etc. During the testing phase, one of the datasets is selected for inference to evaluate the performance of the model under original conditions and under attacked conditions, respectively. The evaluation indicators include mean average accuracy (mAP) and Rank-1 recognition rate, with higher values ​​indicating stronger model robustness.

Images