Pedestrian re-identification method based on vision transformer attention screening

Abstract

The application discloses a pedestrian re-identification method based on VisionTransformer attention screening. The pedestrian features that are focused on are obtained according to attention response values to help model training and identification. The application comprises the following steps: 1. obtaining pedestrian images with pedestrian identity labels, performing image enhancement operations on the images, and uniformly processing the images to ensure that the model can receive the input of the image set; 2. constructing a pedestrian re-identification model based on global-local feature extraction of Vision Transformer; 3. training the pedestrian re-identification model until the model converges; 4. using a test set to evaluate the model generalization performance of the trained model. Compared with random shuffling of features, the attention screening used in the application can better help model training, improve the discrimination ability of the model, reduce the influence of interference features on model identification, and improve the robustness of the model.

Description

Technical Field

[0001] This invention utilizes the Vision Transformer deep learning model pre-trained on ImageNet, and employs a series of improvements and methods to help the model better identify pedestrians for the downstream task involved in this invention—person re-identification. The person re-identification task is a sub-task of image classification in computer vision. Background Technology

[0002] In recent years, with the rapid development of intelligent surveillance equipment and the increasing demand for public safety, a large number of cameras have been deployed in public places such as airports, communities, streets, and campuses. These camera networks typically span large geographical areas with non-overlapping coverage, generating a large amount of surveillance video every day. We use this video data to analyze the activity patterns and behavioral characteristics of pedestrians in the real world for applications such as target detection, multi-camera target tracking, and crowd behavior analysis. Person Re-ID traces back to the multi-target multi-camera tracking (MTMCT) problem, aiming to determine whether pedestrian images captured by different cameras or different video clips from the same camera belong to the same pedestrian. Person Re-ID constructs a large image dataset (Gallery) from the detected pedestrian images and uses probe images (queries) to retrieve matching pedestrian images from it; therefore, Person Re-ID can also be viewed as an image retrieval task. The key to Person Re-ID is learning the discriminative features of pedestrians to distinguish between pedestrian images with the same identity and those with different identities. However, in the real world, pedestrians may appear in multiple cameras in multiple areas, and the variations in the perspective, posture, lighting and resolution of different cameras increase the difficulty of learning pedestrian recognition features.

[0003] Traditional pedestrian re-identification methods mainly rely on manually extracting fixed discriminative features or learning better similarity measures. These methods are prone to errors and are extremely time-consuming, significantly impacting the accuracy and real-time performance of pedestrian re-identification tasks. In 2014, deep learning was first applied to the field of human re-identification.

[0004] Before the advent of the Vision Transformer, various CNN-based methods had achieved great success in the field of pedestrian re-identification. After the Vision Transformer achieved further success in NLP and many other visual fields, researchers began to focus on designing and improving methods to address the challenges of pedestrian re-identification. Compared to CNN-based methods, Vision Transformer-based methods exhibit the following advantages:

[0005] 1. Since the Vision Transformer structure does not include the downsampling operation found in CNN methods, this is a significant change for the current low image resolution of pedestrian re-identification datasets. Preserving complete detail information makes it easier for the model to identify pedestrian images in special scenarios, such as when different pedestrians are wearing clothing of similar colors.

[0006] 2. Compared to traditional CNN models, which use convolutional operations and thus have a limited receptive field, failing to capture the dependencies between two distant parts of an image and thus hindering the ability to focus on various parts of the human body, the VisionTransformer-based method addresses these shortcomings through its global multi-head attention.

[0007] Currently, due to the specific nature of pedestrian re-identification datasets and the high cost of manual data annotation, large-scale, realistic, and complex datasets are lacking, thus failing to fully realize the upper limit performance of the Vision Transformer. Furthermore, in some cases, overfitting can occur, resulting in the model lacking generalization ability, and visualizations that over-focus on a particular salient part of the pedestrian. Summary of the Invention

[0008] This invention addresses the shortcomings of current Vision Transformer-based person re-identification methods by proposing an attention-based selection method. By performing attention-based selection on features with high-level semantic information, regions containing highly attentionable features can be better integrated to aid model training and improve recognition accuracy. Simultaneously, this method segments and encodes the selected features, obtains relevant prediction scores through a classifier, and calculates their losses. When assigning loss weights, a relatively small value should be allocated to the loss obtained from the fourth part of the features. This allows the model to reduce interference from other features when identifying salient pedestrian features and better mine fine-grained features to aid in identification.

[0009] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0010] Step (1): Obtain pedestrian images with pedestrian identity tags, perform image enhancement operations on the images, and uniformly process the images to ensure that the model can receive the input of the image set.

[0011] Step (2): Construct a pedestrian re-identification model based on global-local feature extraction using Vision Transformer. All preprocessed pedestrian images obtain pedestrian features through the two branches (global branch and local branch) of the backbone model in the pedestrian re-identification model, and obtain pedestrian identity predictions through multiple classifiers.

[0012] Step (3): Train the pedestrian re-identification model until the model converges. Calculate the loss using the obtained pedestrian identity predictions, pedestrian features output by the backbone model in the pedestrian re-identification model, and pedestrian identity labels. Then use stochastic gradient descent to reduce the loss and help the model converge.

[0013] Step (4) Pedestrian Prediction. Obtain multiple sets of pedestrian images containing pedestrian identity labels that were not used in the training process. Each pedestrian should have at least two images. After uniformly processing the image sets, extract one image from each pedestrian image as the query, and the remaining images as the gallery. Input the query and gallery into the model to obtain two sets of features, calculate the distance between these two sets of features, and find the image in the gallery that is most similar to the query.

[0014] Furthermore, the specific implementation process of step (1) is as follows:

[0015] Prepare an image dataset containing pedestrian identification labels. This dataset was obtained through three methods: pedestrian detection model, generative model, and manual annotation. Each image should contain a complete image of a pedestrian from the camera's perspective. First, perform uniform enhancement processing on the images: random horizontal flipping, padding, random partial cropping, random erasing, and regularization. Then, crop the images to a length and height of 256*128. Before inputting the images into the model, ensure sample balance; that is, each batch of input images should have the same number of samples for each pedestrian class. Empirically, this can be set to 4-8 images. If the original number of pedestrian images is insufficient, randomly copy them; otherwise, randomly select images.

[0016] Furthermore, the specific implementation process of step (2) is as follows:

[0017] A pedestrian re-identification model based on Vision Transformer attention filtering is constructed. First, a Transformer block is built, which consists of a multi-head self-attention method (MSA), layer normalization (LN), and a multilayer perceptron (MLP). The formula for a single Transformer block is:

[0018] Z′ l =MSA(LN(Z) l-1 ))+Z l-1 l=1…L (1)

[0019] Z l =MLP(LN(Z′) l ))+Z′ l l=1…L (2)

[0020] The Transformer as a whole consists of L layers of Transformer blocks, where Z... l Let L represent the features output by the l-th Transformer block, where L is 12. The multi-head self-attention method calculates the similarity within the features, as shown in the following formula:

[0021]

[0022] Attention is addressed by calculating the similarity between Q and K. The similarity between Q and K reflects the importance of the retrieved V values, i.e., the weights, and then a weighted sum is obtained to obtain the attention value. The special feature of the self-attention mechanism in the KQV model is that K = Q = V, which represent the query, key, and value in the attention mechanism process, respectively. This is why it is called a self-attention mechanism. k These are the dimensions of the outputs of Q(*) and K(*). This is a scaling factor. The invention employs a multi-head self-attention mechanism, aiming to allow each head to adaptively learn different attention weights, thereby improving the robustness of the overall structure. The specific implementations of K, Q, and V are as follows:

[0023]

[0024] Before inputting an image into a Transformer block for self-attention computation, each image needs to be divided into multiple patches (patch embeddings); for example, if an image is 256*128*3, it needs to be segmented into: Where P is 16, which is the side length of the patch.

[0025] To avoid disrupting the correlation between patches, non-overlapping patch embedding is used, as shown in the following formula:

[0026]

[0027] Where S is the stride, H and W are the length and width of the image, and N is the number of patches obtained by non-overlapping segmentation.

[0028] The processed tensor, along with the initialized [CLS_TOKEN] and [POS_TOKEN], is input into the model. The specific formula is as follows:

[0029]

[0030] Where, x cls It is [CLS_TOKEN], used for the final classification prediction; It is a feature after patchwork bedding is completed; E pos [POS_TOKEN] represents the spatial location and is used to preserve the relative positional relationship between patches.

[0031] Features Out1 and Out2 are obtained through the penultimate and last Transformer blocks of the model, respectively. Out2 is the pedestrian feature output by the backbone model in the pedestrian re-identification model, which is then used as a global feature for subsequent global loss calculation.

[0032] Features Out1 In [CLS_TOKEN] Separation, separation The feature representation after [CLS_TOKEN] is Out', which is filtered using an attention-based method, specifically:

[0033] 1. Get all input feature Out' represents all the maximum values ​​in the last dimension;

[0034] 2. Obtain the sorted index based on the maximum value of the last dimension;

[0035] 3. Sort all inputs according to the sort index. feature The `Out` option is used to sort and reorganize the data.

[0036] Set up a copy block, copy the last Transformer block with the same initialization parameters, and evenly divide the recombined features after sorting and rearranging into four parts according to the last second dimension. Then, respectively, they are compared with... feature Out1 Separated from [CLS_TOKEN] is combined to obtain the final local features by copying the block.

[0037] Furthermore, the specific implementation process of step (3) is as follows:

[0038] The labels from the pedestrian re-identification dataset are used as supervision information, and together with global and local features, a difficult triplet loss is calculated to aid network training. Global and local features are then passed through a classifier to obtain prediction scores, and cross-entropy loss is used to further train the model. The specific formula for cross-entropy loss is as follows:

[0039]

[0040] Where N is the number of pedestrian IDs, q i It is supervisory information, p i It is the predicted score.

[0041] The hard triplet loss method involves randomly sampling P identities, extracting K instances from each identity, and forming a minibatch of size P*K. Images within the batch are then selected as anchor points, and the farthest positive sample and the nearest negative sample are chosen to form triplet training networks. The specific formula is as follows:

[0042]

[0043] Among them, f a It is an anchor point sample, f p f n These are the corresponding positive and negative samples.

[0044] In the local branch, attention filtering divides the overall features into multiple local features. In the case of equal segmentation, the final features after segmentation should be secondary and irrelevant features of the pedestrian. When classifying and predicting this part, its confidence level should be "unreliable." However, the existence of these secondary features can help the model identify individuals with similar overall appearance features, which is beneficial for mining fine-grained features. Therefore, a relatively small value should be assigned to the loss weight in this part. The specific formula is as follows:

[0045]

[0046]

[0047] Among them, L id (f) and L T (f) represent the cross-entropy loss and the hard triplet loss calculated on the features, respectively; f global and f local These represent global features and local features, respectively; k is the number of local features, with a value of 4; λ is the loss weight.

[0048] Furthermore, the specific implementation process of step (4) is as follows:

[0049] After obtaining a stable model, acquire a pedestrian re-identification dataset that has not been used in the training process, requiring at least two images for each pedestrian. After uniformly processing all images, input them into the model to obtain global and local features. The global features need to be concatenated with the first three local features, using the following formula:

[0050] f test =[fglobal ;f local_1 ;f local_2 ;f local_3 |dim=1] (10)

[0051] Calculate the Euclidean distance between the corresponding features of all query and gallery images to obtain the most similar pedestrian images.

[0052] The beneficial effects of this invention are as follows:

[0053] This invention proposes a pedestrian re-identification method based on Vision Transformer attention filtering. Compared to CNN models, Transformer-based models can preserve more detailed information, which is extremely useful in the field of pedestrian re-identification. Extensive experiments have shown that the upper limit of Transformer models is significantly higher than that of CNN models, which is beneficial for improving pedestrian re-identification performance. Using a global-local dual-branch structure helps the model focus on various parts of the human body, rather than the most prominent parts, reducing overfitting problems caused by insufficient data. The attention filtering method can acquire multiple key features while simultaneously utilizing secondary features to mine fine-grained information from the image, improving the model's robustness. Attached Figure Description

[0054] Figure 1 This is a complete flowchart of the present invention. Detailed Implementation

[0055] The detailed parameters of the present invention will be further described below.

[0056] This invention provides a deep neural network model for supervised image pedestrian re-identification. Figure 1 This is a flowchart illustrating the overall implementation of the present invention.

[0057] In step (1), select the mainstream artificial datasets MSMT17 and Market1501, and choose the training set and corresponding pedestrian identity labels. Since the resolution of the datasets is not completely uniform, it is conventionally set to 256*128, and then the images need to be randomly augmented. Since the number of pedestrian images contained in each identity is not exactly the same, it is necessary to ensure that the number of pedestrian images for each identity is consistent in each training batch. For example, in a batch of pedestrian image data with a size of 64, 16 identities can be randomly selected, and each identity should have 4 images. If there are less than 4 images in the original dataset, they are obtained by random copying; otherwise, they are obtained by random selection.

[0058] In step (2), the base version of Vision Transformer is selected as the base model, and the model parameters pre-trained on the ImageNet dataset are loaded. The last Transformer block is copied for encoding local features, and multiple batch regularization blocks and classifiers are initialized for pedestrian prediction. Due to the limitations of the number of GPUs and video memory, distributed training (DDP) and mixed precision training (FP16) in the PyTorch framework are chosen to help reduce training memory usage and improve training speed.

[0059] 2-1. Each batch of images is input into the image and overlay patch embedding is performed according to the formula:

[0060]

[0061] When S is 16, it represents the non-overlapping patch embedding typically used by Transformers. This invention sets S = 12. The patch, [CLS_TOKEN], and [POS_TOKEN] are combined to obtain the final features used in training, as shown in the following formula:

[0062]

[0063] The above features are then processed through 11 Transformer blocks to obtain feature Z. 11 The specific formula is as follows:

[0064] Z′ l =MSA(LN(Z) l-1 ))+Z l-1 l = 1…L

[0065] Z l =MLP(LN(Z′) l ))+Z′ l l = 1…L

[0066] 2-2, Z 11 The final global features are obtained directly from the last original Transformer block. These final global features are then processed through batch regularization and a classifier to obtain a prediction score for the global features. Both the global features and the global prediction score will be used for subsequent model training.

[0067] 2-3, Z 11The model is separated into a new [CLS-TOKEN] and features. The features are then segmented using an attention-based selection method to obtain multiple sets of local features. These are then combined with the new [CLS-TOKEN] and processed in the same manner using a copy block, a batch regularization block, and a classifier to obtain multiple sets of local features and multiple local classification scores. Ideally, the distribution of prediction scores for each classifier should be similar or identical. The obtained features and prediction scores will be used in the same way during model training.

[0068] In step (3), the final global and local features are used as supervision information with the labels from the person re-identification dataset. The hard triplet loss is calculated to help train the network. The two types of features are passed through a classifier to obtain predicted scores, and cross-entropy loss is used to help train the model. The specific formula for cross-entropy loss is as follows:

[0069]

[0070] N is the number of pedestrian IDs, q i It is a supervisory signal, p i This is a predicted value.

[0071] The hard triplet loss method involves randomly sampling P identities, extracting K instances from each identity, and forming a minibatch of size P*K. Images within the batch are then selected as anchor points, and the farthest positive sample and the nearest negative sample are chosen to form triplet pairs to train the network. The specific formula is as follows:

[0072]

[0073] In the local branch, attention filtering divides the overall features into multiple local features. In the case of equal segmentation, the final features after segmentation should be secondary and irrelevant features of the pedestrian. When classifying and predicting this part, its confidence level should be "unreliable." However, the existence of these secondary features can help the model identify individuals with similar overall appearance features, which is beneficial for mining fine-grained features. Therefore, a relatively small value should be assigned to the loss weight in this part. The specific formula is as follows:

[0074]

[0075]

[0076] Step (4) involves testing the model using the test sets of MSMT17 and Market1501. The specific process is as follows:

[0077] 4-1. All images are uniformly cropped and regularized. Since the number of test sets is relatively small, a single GPU is used and the batch size is set to 256 to speed up training.

[0078] 4-2. Load the trained model parameters and use the model to obtain global and local features from all query and gallery images. Combine the respective global features with the first three sets of local features, using the following formula:

[0079] f test =[f global ;f local_1 ;f local_2 ;f local_3 |dim=1]

[0080] Both query and gallery can obtain the corresponding f. test It uses Euclidean distance to calculate distance and finds the image in the gallery that is most similar to all query images.

[0081] This invention proposes a person re-identification method based on VisionTransformer attention filtering. Compared to CNN models, Transformer-based models can preserve more detailed information, which is extremely useful in the field of person re-identification. Extensive experiments have shown that the upper limit of Transformer models is significantly higher than that of CNN models, which is beneficial for improving person re-identification performance. Using a global-local dual-branch structure helps the model focus on various parts of the human body, rather than the most prominent parts, reducing overfitting problems caused by insufficient data. The attention filtering method can acquire multiple key features while simultaneously utilizing secondary features to mine fine-grained information from the image, improving the model's robustness.

[0082]

[0083] Experiments show that the present invention does indeed have a certain improvement effect.

Claims

1. A pedestrian re-identification method based on Vision Transformer attention screening, characterized in that Comprise the following steps: Step (1), obtaining pedestrian images with pedestrian identity labels, performing image enhancement operations on the pedestrian images, and uniformly processing the pedestrian images to ensure that the model can receive the input of the image set; Step (2), constructing a pedestrian re-identification model based on global-local feature extraction of Vision Transformer; all preprocessed pedestrian images obtain pedestrian features through two branches of the backbone model in the pedestrian re-identification model, and obtain pedestrian identity prediction through multiple classifiers; Step (3), training the pedestrian re-identification model until the model converges; calculating the loss of the obtained pedestrian identity prediction, the pedestrian features output by the backbone model in the pedestrian re-identification model, and the pedestrian identity label, and then using stochastic gradient descent to reduce the loss to help the model converge; Step (4), pedestrian prediction; Obtain multiple groups of pedestrian images containing pedestrian identity labels that do not participate in training, and each pedestrian has at least two images; after uniform processing of the image set, extract one image from each pedestrian corresponding pedestrian image as query, and the remaining images as gallery; input the query and gallery into the model to obtain two groups of features, calculate the distance between the two groups of features, and find the most similar image in the gallery to the query; The specific implementation process of step (3) is as follows: The label of the pedestrian re-identification dataset is used as supervision information to calculate the hard triplet loss together with the global feature and the local feature to help train the network; the global feature and the local feature are respectively passed through the classifier to obtain the prediction score, and the cross-entropy loss is used to help train the model; the cross-entropy loss formula is as follows: where N is the number of pedestrian IDs, q i is the supervision information, p i is the prediction score; The hard triplet loss is obtained by randomly sampling P identities, and K instances are extracted from each identity to form a mini batch with a size of P*K; the pictures in the batch are selected as anchor points in turn, and the farthest positive sample and the closest negative sample are selected to form a triplet to train the network, and the specific formula is as follows: where f a is the anchor sample, f p , f n are the corresponding positive and negative samples; In the local branch, the whole feature is divided into multiple local features after screening by attention, and in the case of equal segmentation, the last feature after segmentation is the secondary feature and irrelevant feature of the pedestrian; when the last feature is classified and predicted, its confidence is "unreliable", and the secondary feature may help the model identify individuals with similar overall appearance, which is beneficial to the mining of fine-grained features, so the last feature loss weight should be allocated a relatively small value; the specific formula is as follows: where L id (f) and L T (f) represent the cross-entropy loss and the hard triplet loss for feature computation, respectively; f global and f local represent the global feature and the local feature, respectively; k is the number of local features; and λ is the loss weight.

2. The Vision Transformer attention-based screening method for pedestrian re-identification according to claim 1, characterized in that Step (1) is as follows: Prepare an image dataset containing pedestrian identity labels, which is obtained by three ways of pedestrian detection model, generative model and manual annotation; each image should contain a complete image of a pedestrian in the camera view; first, uniformly enhance the image: random horizontal flip, padding, random partial cropping, random erasing, regularization, and then crop the image into an image with a length of 256*128; before inputting into the model, it is necessary to ensure sample balance, that is, the number of samples of each type of pedestrian in each batch of input images should be the same to ensure that the model does not converge in a suboptimal solution.

Images