Re-identification method based on cross self-distillation transformer re-identification network

Abstract

The present application relates to the technical field of visual image recognition, in particular to a re-identification method based on a cross self-distillation Transformer re-identification network, comprising constructing a cross self-distillation Transformer re-identification network and establishing a training set and a test set, inputting the preprocessed training set images into the network for training; the test set after data enhancement processing is divided into query and gallery two parts and input into the ViT basic network in the trained cross self-distillation Transformer re-identification network, the image features are obtained, the similarity calculation and result evaluation are carried out, and the online available network is obtained; the present application also simultaneously provides a method for using the online network. The present application combines knowledge distillation and cross self-attention calculation, strengthens the connection of different input samples, introduces the knowledge distillation method, so that only the output features of the ViT basic network can also achieve the feature effect obtained by the cross self-attention module.

Description

Technical Field

[0001] This invention relates to the field of visual image recognition technology, specifically a re-recognition method based on a cross-self-distillation Transformer re-recognition network. Background Technology

[0002] Re-identification is a technique that uses computer vision to determine the presence of a specific target in an image or video sequence. It is widely considered a sub-problem of image retrieval. Given an image of a monitored target (pedestrian or vehicle), the task is to retrieve images of that target from a set of target images, often across different devices. Because the detected images and video sequences and the target pedestrian images often come from different times or cameras, there are significant differences in the angle, lighting, and environment of the people or objects in the images. This results in images of the same person or object having lower similarity than images of different people or objects. In other words, after feature extraction, the features of images of the same person or object cluster according to different cameras, and the similarity between different clusters is low, similar to the effect of domain gaps in transfer learning.

[0003] The ViT (Vision Transformer) model decomposes the input image into multiple sequences using convolution operations with the same stride and kernel size. Positional information is added, and these sequences are then concatenated. An initial [CLS] sequence of the same size as the sequence is added. Self-attention is then used to calculate mutual attention between each sequence, obtaining the mutual information between them. After multiple self-attention calculations, the [CLS] sequence or all sequences can be used as features for subsequent tasks. In recent years, ViT has achieved significant success in computer vision due to its superior performance, and more and more fields are using ViT networks as base models, including re-identification. However, ViT does not optimize for the core problem of re-identification: the different features of the same pedestrian in different external environments and cameras. Currently, widely used methods use Arcface Loss and Triplet Loss as constraints, and some researchers have improved this method by focusing on model structure. Due to differences in external environments such as lighting, occlusion, and different camera parameters, the feature spacing of the same pedestrian extracted by the model may be larger than that of different pedestrians.

[0004] The MIT-IBM Watson AI Lab paper (CrossViT: Cross-Attention Multi-ScaleVision Transformer For Image classification) proposes a ViT model that combines a cross-attention module to perform target recognition on image patches of different sizes in order to generate stronger image features. However, the cross inputs of this method are different modalities, different scales, or different parts of the same input image. The purpose is to learn the features of the input image better, but it cannot perform cross-self-attention calculation on the features of images from different domains formed by different cameras.

[0005] Therefore, existing technologies need to be improved. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a re-identification method based on a cross-self-distillation Transformer re-identification network for re-identifying target objects in images obtained from different cameras or angles.

[0007] To address the aforementioned technical problems, this invention provides a re-identification method based on a cross-self-distillation Transformer re-identification network, comprising the following steps:

[0008] S1. Using the ViT network as the base network, a cross-self-distillation Transformer re-identification network is constructed by adopting a cross-self-distillation structure between the ViT base network and the cross-attention module.

[0009] S2. Preprocess the images in the training set and perform data augmentation on the images in the test set;

[0010] S3. Input the preprocessed training set images into the cross-self-distillation Transformer re-identification network for training;

[0011] S4. Divide the data augmentation-processed test set into two parts, query and gallery, and input them into the ViT base network in the cross-self-distillation Transformer re-identification network trained in step S3. Obtain the features of each image in query and gallery, perform similarity calculation and result evaluation to obtain the cross-self-distillation Transformer re-identification network that can be used online.

[0012] S5. When using online, the input image includes the image of the target ID and the image set to be retrieved. After data augmentation processing of the image of the target ID and the image set to be retrieved, they are respectively input into the ViT base network in the online cross-distillation Transformer re-identification network obtained in step S4 for feature extraction. Then, the similarity between the image features of the target ID and the features of all images in the image set to be retrieved is calculated to obtain the most similar sample with the image features of the target ID, thereby obtaining the re-identification result of the target ID.

[0013] As an improvement to the re-identification method based on the cross-self-distillation Transformer algorithm of this invention:

[0014] The preprocessing of the training set is as follows: after resizing the images in the training set, data augmentation operations of RandomHorizontalFlip, Pad, RandomCrop, and Normalize are performed sequentially. Then, an improved PK-sampling process is applied, specifically: in each ID, there are c samples from each camera out of k samples. If the number of cameras remaining in an ID needs to be... There are several cameras, but the current ID is {v1, v2, ..., v}. r There are only r cameras (where, First, randomly select c samples from each camera, and then select the remaining samples from the cameras {v1, v2, ..., v3} that still have samples. r' Random selection from} From a given camera / viewpoint, select c samples;

[0015] The data augmentation process involves resizing each image and then performing a Normalize operation.

[0016] As a further improvement to the re-identification method based on the cross-self-distillation Transformer algorithm of the present invention:

[0017] The cross-self-distillation Transformer re-identification network first passes the input data through the ViT base network to generate student logits. The student logits are then trained under supervision using the loss functions Arcface Loss and Cross Entropy Loss. After hard positive mining, the student logits generate hard pairs. The student logits and hard pairs are then input together into the cross-self-attention module to generate teacher logits. Finally, the teacher logits and student logits are knowledge distilled using the l2-norm loss function.

[0018] As a further improvement to the re-identification method based on the cross-self-distillation Transformer algorithm of the present invention:

[0019] The cross-self-distillation Transformer re-identification network training involves setting network training parameters, including batch size, weight decay, initial learning rate, and training epochs. Each batch includes 16 IDs, and each ID includes 4 samples from different cameras or viewpoints.

[0020] As a further improvement to the re-identification method based on the cross-self-distillation Transformer algorithm of the present invention:

[0021] The similarity calculation formula is as follows:

[0022] x = (x1, x2, ..., x l )

[0023]

[0024] similarity(x,y)=normalized(x)×normalized(y)

[0025] Where x is the output feature of the query image through the ViT base network, and y is the output feature of the gallery image through the ViT base network.

[0026] The beneficial effects of this invention are mainly reflected in:

[0027] 1. This invention performs cross-self-attention calculation on image features from different domains of different cameras, which to some extent bridges the domain gap in the output features and eliminates the influence of the domain gap caused by different cameras;

[0028] 2. This invention combines knowledge distillation and cross-self-attention computation to strengthen the connection between different input samples. By introducing the knowledge distillation method, it enables the output features of the ViT basic network to achieve the same effect as the features obtained by the cross-self-attention module when used online.

[0029] 3. Based on the characteristics of the cross self-attention module, this invention designs an improved PK-sampling method to avoid the hard pairs obtained by hard positive mining of mini-batch from the same camera. Attached Figure Description

[0030] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0031] Figure 1 This is a schematic diagram of the cross-self-distillation Transformer re-identification network of the present invention;

[0032] Figure 2 for Figure 1 A schematic diagram of the structure of the basic ViT network;

[0033] Figure 3 for Figure 1 A schematic diagram of the Cross-attention module in the middle section;

[0034] Figure 4 This is a schematic diagram of the structure of a single-layer Transformer module. Detailed Implementation

[0035] The present invention will be further described below with reference to specific embodiments, but the scope of protection of the present invention is not limited thereto:

[0036] Example 1: A re-identification method based on a cross-self-distillation Transformer re-identification network, such as... Figure 1-4 As shown, it includes the following steps:

[0037] S1. Construct a cross-self-distillation Transformer re-identification network, including a ViT base network (Backbone) and a cross-attention module, and adopt a cross-self-distillation structure between the ViT base network (Backbone) and the cross-attention module (Cross-attention Module);

[0038] The S101 and ViT basic network consists of an input embedding layer (Patch Embed) and multiple Transformer layers, using a 12-layer structure, with each Transformer layer having the same structure. The input embedding layer transforms the input image into multiple one-dimensional sequences through convolution, adds positional information, concatenates them, and adds an initialization [CLS] sequence of the same size as the sequence. Then, the extracted features are obtained through multiple Transformer layers, such as... Figure 2 As shown.

[0039] The specific structural order of the ViT basic network is as follows:

[0040] The input embedding layer consists of a single convolutional layer (conv) with the same stride and kernel size, and the same number and parameters of kernels. After the convolutional layer, a function concatenates the one-dimensional sequence and initializes a [CLS] sequence of the same size as the first dimension. Position embedding is set as learnable parameters added to the concatenated sequence.

[0041] The Transformer layer contains two residual blocks. The first residual block first performs layer normalization on the input, and then performs a self-attention operation, as shown in the following formula.

[0042]

[0043] Q, K, and V are the results of multiplying the same input with the corresponding parameter matrix. A multi-head attention algorithm is used, which involves dividing each one-dimensional sequence X into equal parts (X is 12 here) and performing the attention calculations. After the calculations are completed, the sequences are then reassembled.

[0044] The second residual structure performs layer normalization on the input, followed by two fully connected layers for dimensional scaling. The first fully connected layer is followed by a Gelu activation function, and the second fully connected layer is followed by a Dropout layer. Its main goal is to obtain more hidden layer information through dimensional scaling. Its overall structure can be represented by the following formula.

[0045]

[0046] The output features of the ViT base network are trained under supervision using the loss functions Arcface Loss and Cross Entropy Loss.

[0047] S102. Combine the Cross-attention Module with the ViT basic network to calculate the cross information of the same pedestrian from different cameras, so as to generate more comprehensive and effective features.

[0048] Hard positive mining finds the samples with the lowest similarity to each sample in a batch, forming a hard pair.

[0049] x hardpair =min(simliarity(x,x) c (4)

[0050] Where x c This represents a sample that belongs to the same pedestrian as x.

[0051] The cross-attention layer consists of four layers, and its structure is the same as that of the Transformer layer. The difference lies in the calculation of self-attention. In equation (1), Q and V are still obtained by multiplying the input x and the parameter matrix, while K comes from x. hardpair The result of calculating the parameter matrix. Therefore, Self-attention, which calculates information between different sequences within the same input, is changed to Cross-attention, which calculates information between sequences from different inputs.

[0052]

[0053] Furthermore, the initialization parameters for these four cross-attention layers are the same as those for the last four layers of the ViT infrastructure, in order to achieve better learning results.

[0054] The challenge of cross-attention modules lies in handling the asynchrony between training and testing, and the issue of obtaining Cross-attention results during testing. Cross-attention strictly requires input samples to belong to the same class, and this operation is only possible when the samples are labeled. In the unlabeled testing phase, it's impossible to obtain such a hard pair of samples. The solution is a cross-self-distillation structure. This structure compares the output features (student logits) of the ViT base network with the output features (teacher logits) after the cross-attention layer, using an L2-norm loss function to constrain the student logits to be close to the teacher logits. In testing and practical applications, only the student logits are used as features.

[0055] After embedding the cross-attention module into the ViT base network and adding a cross-self-distillation structure, the resulting cross-self-distillation Transformer re-identification network is shown in the following structure. Figure 1 As shown, the input data first passes through the base model, the ViT network, to generate student logits. These features are then trained under supervision using Arcface Loss and Cross Entropy Loss. After hard positive mining, the student logits generate hard pairs. These student logits and hard pairs are then fed into a cross-attention module to generate teacher logits. Finally, the teacher logits and student logits undergo knowledge distillation using the l2-norm loss function.

[0056] S2. Train the cross-self-distillation Transformer re-identification network;

[0057] S201, Use of the dataset;

[0058] The datasets used in the experiments were from the publicly available online datasets MSMT17 and Veri-776. The former is a pedestrian re-identification dataset, and the latter is a vehicle re-identification dataset; both are widely used and recognized public datasets in research. The MSMT17 dataset uses a network of 15 cameras, including 12 outdoor cameras and 3 indoor cameras. Four days with different weather conditions were selected, covering the morning, noon, and afternoon time periods. The Veri-776 dataset images were taken by 20 cameras, with each vehicle captured by 2–18 cameras under different viewpoints, lighting, resolutions, and occlusions. Specific dataset information is shown in the table below:

[0059] Table 1

[0060] Dataset MSMT17 Veri-776 Number of pedestrian / vehicle IDs 4101 769 Number of images 126441 37778 Number of cameras 15 20

[0061] The publicly available datasets MSMT17 and Veri-776 have been pre-divided into training and testing sets. The images in both the training and testing sets have been resized to 256*128 pixels. Then, data augmentation operations of RandomHorizontalFlip, Pad, RandomCrop, and Normalize were applied sequentially to each image in the training set, while only the Normalize operation was applied to each image in the testing set.

[0062] S202, Preprocessing of the training set.

[0063] Because hard sample mining is required in the model, each input batch needs to contain multiple samples with the same ID. PK-sampling is a widely used sampling method. PK-sampling refers to randomly sampling each mini-batch, which includes p IDs, with k samples for each ID (each ID corresponds to a person or a vehicle). However, in re-identification tasks, different cameras and viewpoints result in images from different domains, leading to domain gaps. To better complement hard sample mining and cross-distillation modules and reduce domain gaps across cameras and viewpoints, PK-sampling needs to be improved. Specifically, for each ID, each camera has c samples out of the k samples. If the number of cameras retaining a particular ID needs to be increased... There are several cameras, but the current ID is {v1, v2, ..., v}. r There are only r cameras (where, First, randomly select c samples from each camera, and then select the remaining samples from the cameras {v1, v2, ..., v3} that still have samples. r' Random selection from} We select c samples from each camera / viewpoint to ensure that each camera has c samples from the k samples in each ID as input for network training, thus avoiding hard pairs obtained by hard positive mining from the mini-batch from the same camera.

[0064] S203, Training Environment Configuration.

[0065] The training of the cross-self-distillation Transformer re-identification network constructed in step S1 was performed on a CentOS operating system server, using a Tesla P4 GPU for acceleration. The development environment was based on the PyTorch deep learning framework, and the specific hardware and software configurations are shown in the table below:

[0066] Table 2

[0067] name Environment configuration operating system CentOS 7.3.1611 processor 12*E5-2609v3@1.9GHz, 15M Cache graphics card Tesla P4 8GB (384.81) Memory 125GB Development Environment Python 3.7 and PyTorch 1.4.0

[0068] S204, Training Process.

[0069] After undergoing resizing, data augmentation, and preprocessing in steps S201 and S202, the training set data is input into the cross-self-distilling Transformer re-identification network constructed in step 1 for training. The model is trained using the SGD optimizer with a batch size of 64 and a weight decay of 0.0001. Each batch contains 16 IDs, and each ID includes 4 samples from different cameras or viewpoints. The initial learning rate is set to 0.008, and a cosine learning rate decay function is used. The model is trained for a total of 120 epochs, resulting in the trained cross-self-distilling Transformer re-identification network.

[0070] S3. The actual use of the cross-self-distillation Transformer re-identification network model includes the following steps:

[0071] The input images include the image of the target ID (query) and all images to be retrieved (gallery). The image size is adjusted to 256*128 and processed by normalization. Then, the images are input into the ViT base network of the cross-self-distillation Transformer re-identification network trained in step S203 for feature extraction. The image features of the target ID and the features of each image in the gallery are obtained respectively. Then, the image features of the target ID and the features of all images in the gallery are used to calculate the similarity using the cosine similarity method to obtain the most similar sample with the image features of the target ID, thereby obtaining the re-identification result of the target ID.

[0072] ViT is a fundamental application model of the Transformer model with self-attention as its core mechanism in the field of computer vision. In recent years, algorithms based on ViT have achieved great success in various fields of computer vision. However, ViT is primarily designed for image classification applications. Furthermore, ViT does not optimize for the core problem of re-identification tasks: the different features of the same pedestrian in different external environments and cameras. Therefore, directly applying ViT to re-identification tasks will result in performance deficiencies. This invention proposes a cross-self-distillation Transformer re-identification network. This network uses a cross-attention algorithm to reduce the influence of domain gaps between different samples from different cameras, extracting image features closer to the class center. Knowledge distillation further enables the features extracted by the ViT-based network to achieve the same effect as cross-attention feature extraction. This method allows ViT-based algorithms to handle the domain gaps between different cameras in re-identification tasks. It also makes the image features extracted directly by ViT more suitable for re-identification scenarios.

[0073] Experiment 1:

[0074] To further verify the capabilities of the proposed method on real-world datasets, this experiment compares the proposed method based on the cross-self-distillation Transformer re-identification network with algorithms from two other related papers on publicly available datasets. Reference 1 (see He S, Luo H, Wang P, et al. Transreid: Transformer-based object re-identification[C] / / Proceedings of the IEEE / CVF international conference on computer vision.2021:15013-15022.) also uses the ViT network as its base network and adds a learnable camera parameter to improve performance by mining local features. Reference 2 (see Quispe R, Lan C, Zeng W, et al. AttributeNet: Attribute enhanced vehicle re-identification[J]. Neurocomputing,2021,465:84-92.) uses the ResNet algorithm as its base network and introduces AttributeNet (ANet) to jointly extract identity-related features and attribute features, effectively exploring discriminative information from vehicle attributes (color and type). The experimental evaluation indicators are as follows:

[0075] 1) Rank-1 Precision

[0076] In classification problems, the model's predictions and the sample's true labels are represented by a confusion matrix to indicate four combinations: True Positive (TP), False Positive (FP), True Negative (TN), and False Negative (FN). The confusion matrix is ​​shown in Table 3 below.

[0077] Table 3

[0078]

[0079] Precision, also known as accuracy, refers to the percentage of samples predicted as positive that are actually positive. It is defined as follows:

[0080]

[0081] In the re-identification problem, for each sample, multiple correct samples may be recalled. Therefore, the precision is generally calculated using the top 1 / 5 / 10 most similar samples, with the calculation method remaining unchanged, resulting in Rank-1 / 5 / 10 Precision respectively. Rank-1 Precision has the highest requirement, and Rank-5 / 10 Precision can be reflected to some extent by mAP, so only Rank-1 Precision is selected as the evaluation metric.

[0082]

[0083] 2) Average Precision (AP) and Mean Average Precision (mAP)

[0084] Average Precision (AP) considers both high precision and high recall. Calculating the AP value for a given sample requires setting a confidence threshold for the prediction results. First, the similarity between each sample's features and the gallery image features is sorted in descending order. Then, positive and negative samples are separated according to the set confidence threshold. Recall samples with a confidence level greater than the threshold are defined as positive samples. For positive samples, the precision and recall coordinates at that confidence threshold are calculated, i.e., a point on the precision-recall (PR) curve. The area enclosed by the curve and the coordinate axes is the AP value for that sample.

[0085] The mean average precision (mAP) is calculated by averaging the AP values ​​of all samples. The value of mAP is always in the range [0, 1], and the larger the value, the better the model performance.

[0086] 3) Davies-Bouldin index (DBI)

[0087] The Davies-Bouldin index (DBI), also known as the classification accuracy index, considers both intra-class and inter-class sample similarity and is typically used to assess the dispersion of clusters. A lower DBI indicates less dispersion. In re-identification problems, different cameras can cause image features to cluster separately. A lower DBI indicates more similar features across different cameras, resulting in smaller domain gaps. The formula for calculating DBI is as follows:

[0088]

[0089] Where f i x is the average of the features of all samples in class i, i.e., the class center. ik Let S be the k-th sample in class i. i Let be the average distance between all samples in class i and their class center. In the DBI calculation of this invention, 1-cosine similarity is used as the distance metric.

[0090] The results of the comparative experiments are shown in Table 4 below:

[0091] Table 4

[0092]

[0093] The experimental parameters for the model in this invention have been given previously. Therefore, for TranReid, which uses the same base model, the results obtained with the same experimental parameters as in this experiment were selected for evaluation in the comparison. For ANET, which uses a different base model, the best set of experimental results was selected. ANET is a vehicle re-identification model, which is difficult to compare in the pedestrian re-identification dataset MSMT17, so the experimental results were set as missing.

[0094] The results show that, compared to the models in the other two references, the model of this invention effectively improves the quality of network feature extraction by adding a module to calculate cross-attention information between samples with the same ID from different cameras. Furthermore, by using a cross-self-distillation structure to transfer the learned cross-attention information to the test, it eliminates the need for hard pairs in practical applications. Therefore, the model of this invention performs better in Rank-1 Precision and mAP metrics.

[0095] Finally, it should be noted that the above examples are merely some specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A re-identification method based on the cross-self-distillation Transformer algorithm, characterized in that, The process includes the following: S1. Using the ViT network as the base network, a cross-self-distillation Transformer re-identification network is constructed by adopting a cross-self-distillation structure between the ViT base network and the cross-attention module. S2. Preprocess the images in the training set and perform data augmentation on the images in the test set; The training set preprocessing involved resizing the images and then sequentially applying data augmentation operations: RandomHorizontalFlip, Pad, RandomCrop, and Normalize. Then, an improved PK-sampling process was used: for each ID, k samples were processed from each camera (c samples). If a camera in an ID needed to be retained... There are several cameras, but the current ID is {v1, v2, ..., v}. r There are only r cameras. First, randomly select c samples from each camera, and then select the remaining samples from the cameras {v1, v2, ..., v3} that still have samples. r' Random selection from} From a given camera / viewpoint, select c samples; The data augmentation process involves resizing each image and then performing a Normalize operation. S3. Input the preprocessed training set images into the cross-self-distillation Transformer re-identification network for training; S4. Divide the data augmentation-processed test set into two parts, query and gallery, and input them into the ViT base network in the cross-self-distillation Transformer re-identification network trained in step S3. Obtain the features of each image in query and gallery, perform similarity calculation and result evaluation to obtain the cross-self-distillation Transformer re-identification network that can be used online. S5. When using online, the input image includes the image of the target ID and the image set to be retrieved. After data augmentation processing of the image of the target ID and the image set to be retrieved, they are respectively input into the ViT base network in the online cross-distillation Transformer re-identification network obtained in step S4 for feature extraction. Then, the similarity between the image features of the target ID and the features of all images in the image set to be retrieved is calculated to obtain the most similar sample with the image features of the target ID, thereby obtaining the re-identification result of the target ID.

2. The re-identification method based on the cross-self-distillation Transformer algorithm according to claim 1, characterized in that: The cross-self-distillation Transformer re-identification network first passes the input data through the ViT base network to generate student logits. The student logits are then trained under supervision using the loss functions Arcface Loss and Cross Entropy Loss. After hard positive mining, the student logits generate hard pairs. The student logits and hard pairs are then input together into the cross-self-attention module to generate teacher logits. Finally, the teacher logits and student logits are knowledge distilled using the l2-norm loss function.

3. The re-identification method based on the cross-self-distillation Transformer algorithm according to claim 2, characterized in that: The cross-self-distillation Transformer re-identification network training involves setting network training parameters, including batch size, weight decay, initial learning rate, and training epochs. Each batch includes 16 IDs, and each ID includes 4 samples from different cameras or viewpoints.

4. The re-identification method based on the cross-self-distillation Transformer algorithm according to claim 3, characterized in that: The similarity calculation formula is as follows: Where x is the output feature of the query image through the ViT base network, and y is the output feature of the gallery image through the ViT base network.

Images