A pedestrian re-identification method based on multi-granularity visual converter

Abstract

The present application belongs to the technical field of computer vision, and proposes a pedestrian re-identification method based on a multi-granularity visual converter. In the present application, three feature extraction stages are added to the visual converter: scattering, segmentation and splicing, purification and strengthening, so that the features learned by the network gradually become from rough to fine. In the strengthening stage of the third stage, a feature enhancement block is added to further enhance the features extracted by the branch. In addition, different loss functions are used in different branches to further improve the network performance. The present application effectively enhances the multi-granularity extraction capability of the visual converter and achieves significant results in the field of pedestrian re-identification.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a pedestrian re-identification method based on a multi-granularity visual converter. Background Technology

[0002] The goal of pedestrian re-identification is to match images of the same person captured from different cameras; hence, it is also known as pedestrian re-identification. In recent years, this technology has been widely used in the field of smart security, such as suspect tracking and missing child retrieval.

[0003] With the explosive popularity of Transformers in the field of vision, people have tried to introduce Transformers into the field of person re-identification. In existing methods, such as "Dosovitskiy A, Beyer L, Kolesnikov A, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale[J].2020.", the image is often simply split into blocks and input into the Transformer, and after processing, a fully connected layer is added to obtain the final result.

[0004] Based on experience exploring convolutional neural networks, dividing an image horizontally into strips and then performing attention weighting and feature extraction on each strip often yields more useful features than extracting the original image. Therefore, existing Transformer models can be further optimized and improved in terms of fine-grained feature extraction. Summary of the Invention

[0005] This invention proposes a pedestrian re-identification method based on a multi-granularity visual converter, aiming to solve the problems mentioned in the background art and improve the model's fine-grained feature extraction capability.

[0006] The technical solution of the present invention is as follows:

[0007] A pedestrian re-identification method based on a multi-granularity visual converter includes the following steps:

[0008] Step 1: Data preprocessing;

[0009] Let N be the number of pedestrian IDs, and let F be the number of pedestrian IDs. i Zhang images, where F i >1; When training the model, pedestrian images are used as input and identity ID information is used as labels; X1% of the images in the training set are used as training images and X2% of the images are used as verification images; The model is a network model based on the existing Vision Transformer with added multi-granularity feature extraction.

[0010] 1.1) Scale the image to (H, W, C) as the input to the model, where H represents the length of the image, W represents the width of the image, and C represents the number of channels in the image;

[0011] 1.2) Data augmentation using random horizontal flipping and random erasure algorithms;

[0012] Set a threshold probability P, generate a random number P1 between 0 and 1. If P1 > P, do not process the input image; otherwise, horizontally flip and randomly erase the input image.

[0013] P1 = Random(0,1)(1)

[0014] During random erasure, the proportion of the erased portion to the overall image is set to [e1-e2], and the aspect ratio of the erased portion is set to R. e ;

[0015] Step two: Process the image format and then input it into the model;

[0016] 2.1) Block embedding;

[0017] Starting from the top left corner of the image, the input image is divided into a set of non-overlapping image patches, each patch having a size of H. block ×W block The image is divided into several segments of size (H). block W block ,3) Image blocks, the number of image blocks N block for:

[0018] N block =(H / H) block )×(W / W block (2)

[0019] Flatten the image patch from two dimensions into one dimension and concatenate them into a one-dimensional vector V. input ;

[0020] 2.2) Add category labels;

[0021] The one-dimensional vector V obtained in step 2.1) input Based on the one-dimensional vector V input The classification identifier token is concatenated at the front end to obtain the final input vector V:

[0022] V = [token, V] input (3)

[0023] 2.3) Input the input vector V into the pre-trained Vision Transformer;

[0024] Input the input vector V obtained in step 2.2) into the Vision Transformer, where the Vision Transformer is pre-trained using the ImageNet public dataset and the Vision Transformer backbone network contains 12 layers of Transformer blocks.

[0025] 2.4) Feature Decomposition

[0026] The vector processed in step 2.3) is split again into a classification identifier token′ and a one-dimensional vector V. input The classification identifier token is used to extract global features; the one-dimensional vector V input Then proceed to step three to perform multi-granularity feature extraction;

[0027] Step 3: Multi-granularity feature extraction stage;

[0028] This invention adds three feature extraction stages to the backend of the Vision Transformer: a fragmentation stage, a segmentation and stitching stage, and a purification and enhancement stage, as detailed below:

[0029] 3.1) The first stage is to break it up;

[0030] Randomly shuffle a one-dimensional vector V input The order of image blocks in the image is used to break them down and blend them, as follows:

[0031] ′′

[0032] V input =shuffle(V input (4)

[0033] 3.2) The second stage is segmentation and splicing;

[0034] 3.2.1) Split

[0035] The one-dimensional vector V after being broken and mixed input Divide the sample into four equal parts from top to bottom, and define the resulting one-dimensional vector V after mixing. input If the length of ′ is L, then V1, V2, V3, and V4 are:

[0036]

[0037]

[0038]

[0039]

[0040] 3.2.2) Splicing

[0041] The category identifier tokens obtained in step 2.4) are concatenated to the beginning of V1, V2, V3, and V4 respectively, as shown in the following expression:

[0042] V1=[token′,V1](9)

[0043] V2=[token′,V2](10)

[0044] V3=[token′,V3](11)

[0045] V4=[token′,V4](12)

[0046] 3.3) The third stage is purification and enhancement;

[0047] 3.3.1) Purification

[0048] The segmented and concatenated feature vectors V1, V2, V3, and V4 are input into the Transformer block for further feature purification. The Transformer block mainly consists of a multi-head self-attention module and a multilayer perceptron. Layer normalization is performed before both, and residual connections are added after both.

[0049] 3.3.2) Strengthen

[0050] The purified feature vectors V1, V2, V3, and V4 are input into the feature enhancement block for feature enhancement; the feature enhancement block consists of three steps: expansion, contraction, and filtering.

[0051] 3.3.2.1) Expansion

[0052] The purified feature vectors V1, V2, V3, and V4 are mapped to an array space that is twice the original length using a dimension-extended fully connected layer.

[0053] 3.3.2.2) Contraction

[0054] The expanded feature vectors V1, V2, V3, and V4 are transformed back into their original length space using a dimension-reduced fully connected layer.

[0055] 3.3.2.3) Filtering

[0056] Using a ReLU layer, negative information obtained from the transformation is discarded, while positive information remains unchanged. The mathematical expression for ReLU is as follows:

[0057]

[0058] Where x represents the eigenvalues ​​in the eigenvectors V1, V2, V3, and V4;

[0059] Step 4: Calculate the loss and backpropagate to update the network parameters;

[0060] 4.1) Feature mapping; The purified and enhanced feature vectors are passed through a fully connected layer to map their features to N, where N is the number of pedestrian types in the dataset set in step one; the features include global features and local features;

[0061] 4.1.1) Global features;

[0062] The classification identifier token′ extracted in step 2.4) is processed through a fully connected layer to obtain the global feature block F. g ;

[0063] 4.1.2) Branching characteristics;

[0064] Extract the classification identifier token′ from the feature vectors V1, V2, V3, and V4 processed in step three, and pass them through fully connected layers to obtain branch feature blocks F. b1 F b2 F b3 and F b4 ;

[0065] 4.2) Use different loss functions for different branches to calculate the loss value;

[0066] 4.2.1) The total loss function includes the global loss and the branch loss, as shown in the following formula:

[0067] L sum =w1L global +w2L branch (14)

[0068] Where w1 and w2 represent the weights of the global loss and branch loss, respectively; L sum L represents the total loss function. global L represents the global loss. branch Indicates branch loss;

[0069] 4.2.2) Add ID loss, Triplet loss, and Circle loss to the global loss, as shown in the following formula:

[0070] L global =f id (F g )+f triplet (F g )+f circle (F g (15)

[0071] Among them, F g The global feature block obtained from step 4.1.1); fid Indicates ID loss; f triplet Indicates Triplet loss; f circle Indicates Circle loss;

[0072] 4.2.3) Add ID loss and Triplet loss to the branch loss, as shown in the following formula:

[0073]

[0074] Among them, F bi The branch feature block obtained from step 4.1.2) includes F b1 F b2 F b3 F b4 Four feature blocks; f id Indicates ID loss; f triplet Indicates Triplet loss;

[0075] 4.3) Set hyperparameters and train the network; use a warm-up learning rate, initially set to r, and gradually increase it tenfold in the first 10 training iterations; use an optimized stochastic gradient descent algorithm as the optimizer, increasing the weight decay by value d1 and the offset by value d2; use the optimized optimizer and learning rate, combined with the loss values ​​calculated in steps 4.1) to 4.2), to perform backpropagation and update the network parameters;

[0076] Step 5: Perform pedestrian re-identification and matching;

[0077] The pedestrian image to be detected is scaled up and then input into the Vision Transformer model with added multi-granularity feature extraction. The output is processed using softmax to obtain N probability values, which correspond to the probability of the pedestrian belonging to different classes. The class with the highest probability value is the identity of the pedestrian.

[0078] The beneficial effects of this invention are as follows: This invention provides a person re-identification method based on a multi-granularity visual converter, adding three feature extraction stages to the visual converter: scattering, segmentation and splicing, and purification and enhancement, allowing the features learned by the network to gradually evolve from coarse to fine. In the third enhancement stage, a feature enhancement block is added to further enhance the features extracted from each branch. Furthermore, different loss functions are used in different branches to further improve network performance. This invention effectively enhances the multi-granularity extraction capability of the visual converter, achieving significant results in the field of person re-identification. Attached Figure Description

[0079] Figure 1 This is a model diagram of a pedestrian re-identification method based on a multi-granularity visual converter according to the present invention; For Circle loss; Triplet loss; For ID loss;

[0080] Figure 2 This is a schematic diagram of the Transformer block.

[0081] Figure 3 This is a schematic diagram of the feature enhancement block of the present invention; As the core feature; ○ represents a useless feature; ○ represents a value of 0. Detailed Implementation

[0082] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings. This embodiment is implemented based on the technical solution of the present invention, and provides detailed implementation methods and specific operation processes. The specific experimental dataset is the Market1501 dataset collected at a university. The training set contains 751 people and 12,936 images; the test set contains 750 people and 19,732 images.

[0083] Figure 1 This is a model diagram of a pedestrian re-identification method based on a multi-granularity visual converter according to the present invention. Figure 2 This is a schematic diagram of the Transformer block. Figure 3 The following is a schematic diagram of the feature enhancement block of the present invention. The specific steps of this embodiment are as follows:

[0084] Step 1: Data preprocessing;

[0085] The training set contains 751 pedestrian IDs, and each ID includes pedestrian F. i Zhang images, where F i >1. When training the model, pedestrian images are used as input, and identity ID information is used as labels; 80% of the images in the training set are used as training images, and 20% are used as verification images;

[0086] 1.1) Scale the image to (384, 128, 3) as the input to the model, where 384 represents the length of the image, 128 represents the width of the image, and 3 represents the number of channels of the image;

[0087] 1.2) Data augmentation using random horizontal flipping and random erasure algorithms;

[0088] Set a threshold probability P = 0.5, generate a random number P1 between 0 and 1. If P1 > P, then no processing is performed on the input image; otherwise, horizontal flipping and random erasing are required.

[0089] P1 = Random(0,1)(1)

[0090] During random erasure, the proportion of the erased portion to the overall image is set to [0.02-0.4], and the aspect ratio of the erased portion is set to 0.3;

[0091] Step two: Process the image format and then input it into the model;

[0092] 2.1) Block embedding;

[0093] Starting from the top left corner of the image, the input image is divided into a set of non-overlapping image patches, where each image patch is 4×4 in size. Therefore, the image is divided into several image patches of size (4,4,3), with the number of patches being N. block for:

[0094] N block = (H / 4) × (W / 4) (2)

[0095] These image patches are flattened from two dimensions to one dimension and then stitched together to form a one-dimensional vector V. input ;

[0096] 2.2) Add category labels;

[0097] The one-dimensional vector V obtained in step 2.1) input Based on the one-dimensional vector V input The classification identifier token is concatenated at the front end to obtain the final input vector V:

[0098] V = [token, V] input (3)

[0099] 2.3) Input the input vector V into the pre-trained Vision Transformer;

[0100] Input the input vector V obtained in step 2.2) into the Vision Transformer, where the Vision Transformer is pre-trained using the ImageNet public dataset and the Vision Transformer backbone network contains 12 layers of Transformer blocks.

[0101] 2.4) Feature Decomposition

[0102] The vector processed in step 2.3) is split again into a classification identifier token′ and a one-dimensional vector V. input The classification identifier token is used to extract global features; the one-dimensional vector V input Then proceed to step three to perform multi-granularity feature extraction;

[0103] Step 3: Multi-granularity feature extraction stage;

[0104] Multi-granularity feature extraction comprises three processing stages, each with its specific purpose, as follows:

[0105] 3.1) Stage 1 (Disperse);

[0106] Randomly shuffle a one-dimensional vector V input The order of image blocks in the image is used to break them down and blend them, as follows:

[0107] V input = shuffle(V) input (4)

[0108] 3.2) Stage 2 (Segmentation and Assembly);

[0109] 3.2.1) Split

[0110] Let the one-dimensional vector V input Divide the sample into four equal parts from top to bottom, and define the resulting one-dimensional vector V after mixing. input If the length of ′ is L, then V1, V2, V3, and V4 are:

[0111]

[0112]

[0113]

[0114]

[0115] 3.2.2) Splicing

[0116] The category identifier tokens obtained in step 2.4) are concatenated to the beginning of V1, V2, V3, and V4 respectively, as shown in the following expression:

[0117] V1=[token′,V1](9)

[0118] V2=[token′,V2](10)

[0119] V3=[token′,V3](11)

[0120] V4=[token′,V4](12)

[0121] 3.3) Stage 3 (Purification and Enhancement);

[0122] 3.3.1) Purification

[0123] The segmented and concatenated feature vectors V1, V2, V3, and V4 are input into the Transformer block for further feature refinement. The Transformer block mainly consists of a multi-head self-attention module and a multilayer perceptron. Layer normalization is performed before both modules, and residual connections are added after both modules.

[0124] 3.3.2) Strengthen

[0125] The purified feature vectors V1, V2, V3 and V4 are input into the feature enhancement block for feature enhancement, which includes three steps: expansion, contraction and filtering.

[0126] 3.3.2.1) Expansion

[0127] The purified feature vectors V1, V2, V3, and V4 are mapped to an array space that is twice the original length using a dimension-extended fully connected layer.

[0128] 3.3.2.2) Contraction

[0129] The expanded feature vectors are transformed back into their original length space using a dimension-reducing fully connected layer.

[0130] 3.3.2.3) Filtering

[0131] Using a ReLU layer, negative information obtained from the transformation is discarded, while positive information remains unchanged. The mathematical expression for ReLU is as follows:

[0132]

[0133] Step 4: Calculate the loss and backpropagate to update the network parameters;

[0134] 4.1) Feature mapping: The purified and enhanced feature vectors are passed through a fully connected layer to map the features to N, where N is the number of pedestrian types in the dataset set in step one.

[0135] 4.1.1) Global features;

[0136] The classification identifier token′ extracted in step 2.4) is processed through a fully connected layer to obtain the global feature block F. g ;

[0137] 4.1.2) Branching characteristics;

[0138] Extract the classification identifier token′ from the feature vectors V1, V2, V3, and V4 processed in step three, and pass them through fully connected layers to obtain branch feature blocks F. b1 F b2 F b3 and F b4 ;

[0139] 4.2) Calculate the loss function;

[0140] 4.2.1) The total loss function consists of two parts: the global loss and the branch loss, as shown in the following formula:

[0141] L sum =w1L global +w2L branch (14)

[0142] Where w1 and w2 represent the weights of the global loss and branch loss, respectively, with w1 taking a value of 0.5 and w2 taking a value of 0.5; L sum L represents the total loss function. global L represents the global loss. branch Indicates branch loss;

[0143] 4.2.2) Add ID loss, Triplet loss, and Circle loss to the global loss, as shown in the following formula:

[0144] L global =f id (F g )+f triplet (F g )+f circle (F g (15)

[0145] Among them, F g The global feature block obtained from step 4.1.1); f id Indicates ID loss; f triplet Indicates Triplet loss; f circle Indicates Circle loss;

[0146] 4.2.3) Add ID loss and Triplet loss to the branch loss, as shown in the following formula:

[0147]

[0148] Among them, F bi The branch feature block obtained from step 4.1.2) includes F b1 F b2 F b3 F b4 Four feature blocks; f id Indicates ID loss; f triplet Indicates Triplet loss;

[0149] 4.3) The hyperparameter settings for training the neural network are shown in Table 1. Using the set optimizer and learning rate, combined with the loss values ​​calculated in steps 4.1) to 4.2), backpropagation is performed to update the network parameters.

[0150] Table 1 Hyperparameter settings for training the network

[0151]

[0152] Step 4: Perform pedestrian re-identification and matching;

[0153] The pedestrian image to be detected is scaled up and input into the model proposed in this invention. After the output is obtained, it is processed by softmax to obtain 751 probability values, which correspond to the probability of the pedestrian belonging to different classes. The class with the highest probability value is the identity of the pedestrian.

[0154] This embodiment tested the pedestrian re-identification performance based on the Market1501 dataset and compared it with existing pedestrian re-identification models, as shown in Table 2:

[0155] Table 2 compares the results with existing baseline models.

[0156]

[0157] The experimental results show that the model proposed in this invention can effectively improve the Rank1 and mAP scores of person re-identification, proving the effectiveness of the method and greatly promoting the practical application of person re-identification.

Claims

1. A pedestrian re-identification method based on a multi-granularity visual converter, characterized in that, Includes the following steps: Step 1: Data preprocessing; Let N be the number of pedestrian IDs, and let F be the number of pedestrian IDs. i Zhang images, where F i >1; When training the model, pedestrian images are used as input and identity ID information is used as labels; X1% of the images in the training set are used as training images and X2% of the images are used as verification images; The model is a network model based on the existing Vision Transformer with added multi-granularity feature extraction; 1.1) Scale the image to (H,W,C) as the input to the model, where H represents the length of the image, W represents the width of the image, and C represents the number of channels in the image; 1.2) Data augmentation using random horizontal flipping and random erasure algorithms; Set a threshold probability P, generate a random number P1 between 0 and 1. If P1 > P, do not process the input image; otherwise, horizontally flip and randomly erase the input image. （1）； During random erasure, the proportion of the erased portion to the overall image is set to [e1-e2], and the aspect ratio of the erased portion is set to R. e ; Step two: Process the image format and then input it into the model; 2.1) Block embedding; Starting from the top left corner of the image, the input image is divided into a set of non-overlapping image patches, each patch having a size of H. block ×W block The image is divided into several segments of size (H). block W block ,3) Image blocks, the number of image blocks N block for: （2）； Flatten the image patch from two dimensions into one dimension and concatenate them into a one-dimensional vector V. input ; 2.2) Add category labels; The one-dimensional vector V obtained in step 2.1) input Based on the one-dimensional vector V input The classification identifier token is concatenated at the front end to obtain the final input vector V: （3）； 2.3) Input the input vector V into the pre-trained Vision Transformer; Input the input vector V obtained in step 2.2) into the Vision Transformer, where the Vision Transformer is pre-trained using the ImageNet public dataset and the Vision Transformer backbone network contains 12 layers of Transformer blocks. 2.4) Feature Decomposition The vector processed in step 2.3) is split again into a classification identifier token′ and a one-dimensional vector V. input The classification identifier token is used to extract global features; the one-dimensional vector V input Then proceed to step three to perform multi-granularity feature extraction; Step 3: Multi-granularity feature extraction stage; Three feature extraction stages have been added to the backend of Vision Transformer: the fragmentation stage, the segmentation and stitching stage, and the purification and enhancement stage, as detailed below: 3.1) The first stage is to break it up; Randomly shuffle a one-dimensional vector V input The order of image blocks in the image is used to break them down and blend them, as follows: （4）； 3.2) The second stage is segmentation and splicing; 3.2.1) Segmentation The one-dimensional vector V after being broken and mixed input Divide the sample into four equal parts from top to bottom, and define the resulting one-dimensional vector V after mixing. input If the length of ′ is L, then V1, V2, V3, and V4 are: （5）； （6）； （7）； （8）； 3.2.2) Splicing The category identifier tokens obtained in step 2.4) are concatenated to the beginning of V1, V2, V3, and V4 respectively, as shown in the following expression: （9）； （10）； （11）； （12）； 3.3) The third stage is purification and enhancement; 3.3.1) Purification The segmented and concatenated feature vectors V1, V2, V3, and V4 are input into the Transformer block for further feature purification. The Transformer block consists of a multi-head self-attention module and a multilayer perceptron. Layer normalization is performed before both, and residual connections are added after both. 3.3.2) Strengthen The purified feature vectors V1, V2, V3, and V4 are input into the feature enhancement block for feature enhancement; the feature enhancement block consists of three steps: expansion, contraction, and filtering. 3.3.2.1) Expansion The purified feature vectors V1, V2, V3, and V4 are mapped to an array space that is twice the original length using a dimension-extended fully connected layer. 3.3.2.2) Contraction The expanded feature vectors V1, V2, V3, and V4 are transformed back into their original length space using a dimension-reduced fully connected layer. 3.3.2.3) Filtering Using a ReLU layer, negative information obtained from the transformation is discarded, while positive information remains unchanged. The mathematical expression for ReLU is as follows: （13）； Where x represents the eigenvalues ​​in the eigenvectors V1, V2, V3, and V4; Step 4: Calculate the loss and backpropagate to update the network parameters; 4.1) Feature mapping: The purified and enhanced feature vectors are passed through a fully connected layer to map their features to N, where N is the number of pedestrian types in the dataset set in step one; the features include global features and local features; 4.1.1) Global features; The classification identifier token′ extracted in step 2.4) is processed through a fully connected layer to obtain the global feature block F. g ; 4.1.2) Branching characteristics; Extract the classification identifier token′ from the feature vectors V1, V2, V3, and V4 processed in step three, and pass them through fully connected layers to obtain branch feature blocks F. b1 F b2 F b3 and F b4 ; 4.2) Use different loss functions for different branches to calculate the loss value; 4.2.1) The total loss function includes the global loss and the branch loss, as shown in the following formula: （14）； Where w1 and w2 represent the weights of the global loss and branch loss, respectively; L sum L represents the total loss function. global L represents the global loss. branch Indicates branch loss; 4.2.2) Add ID loss, Triplet loss, and Circle loss to the global loss, as shown in the following formula: （15）； Among them, F g The global feature block obtained from step 4.1.1); f id Indicates ID loss; f triplet Indicates Triplet loss; f circle Indicates Circle loss; 4.2.3) Add ID loss and Triplet loss to the branch loss, as shown in the following formula: （16）； Among them, F bi The branch feature block obtained from step 4.1.2) includes F b1 F b2 F b3 F b4 Four feature blocks; 4.3) Set hyperparameters and train the network; use a warm-up learning rate, initially set to r, and gradually increase it tenfold in the first 10 training iterations; use an optimized stochastic gradient descent algorithm as the optimizer, increasing the weight decay by value d1 and the offset by value d2; use the optimized optimizer and learning rate, combined with the loss values ​​calculated in steps 4.1) to 4.2), to perform backpropagation and update the network parameters; Step 5: Perform pedestrian re-identification and matching; The pedestrian image to be detected is scaled up and then input into the Vision Transformer model with added multi-granularity feature extraction. The output is processed using softmax to obtain N probability values, which correspond to the probability of the pedestrian belonging to different classes. The class with the highest probability value is the identity of the pedestrian.

Images