Person re-identification method based on twins multi-level features

Abstract

The present invention discloses a pedestrian re-identification method based on twins multi-level features, and the method flow is shown in Figure 1. First, the feature maps of the third and fourth stages of twins are extracted, and the feature map of the third stage is subjected to a designed Convblock, which includes a two-dimensional convolution, a mean normalization layer, an SE attention layer, an activation function layer, and horizontal segmentation to obtain two local feature maps, and then subjected to pooling, dimensionality reduction and other operations to obtain two feature vectors; the feature map of the fourth stage is subjected to pooling, dimensionality reduction and other operations to obtain a feature vector. The three feature vectors are respectively subjected to three classification layers to obtain three predicted classifications. Finally, the feature vector is sent to the triplet loss, and the predicted classification is sent to the cross entropy loss and the center loss for joint training. After 80 iterations of training, the model enters the fitting state, and finally the query set and gallery set of the data set are used to evaluate the performance of the model.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and relates to a multi-branch pedestrian re-identification method based on multi-level feature extraction of Transformer architecture. Technical Background

[0002] Person re-identification, also known as pedestrian re-identification, is a technology that uses computer vision technology to determine whether a specific pedestrian exists in an image or video sequence. It is widely considered to be a sub-problem of image retrieval. Given a monitored pedestrian image, retrieve the pedestrian image across devices. It aims to make up for the visual limitations of fixed cameras and can be combined with pedestrian detection / pedestrian tracking technology. It can be widely used in intelligent video surveillance, intelligent security and other fields. Due to the differences between different camera devices, and the rigidity and flexibility of pedestrians, their appearance is easily affected by clothing, scale, occlusion, posture and perspective, making pedestrian re-identification a hot topic in the field of computer vision that is both valuable and challenging.

[0003] The pedestrian re-identification task mainly includes two steps: feature extraction and similarity measurement. In feature extraction, the Transformer-based backbone model takes into account the correlation of various parts of the input and the low computational cost, and has more choices and possibilities for exploring various subtasks of computer vision. Its vigorous development has brought more powerful feature representation, better resolution and robustness for pedestrian images, which has brought the performance of Re-ID to a new level; in similarity measurement, the unified measurement uses the Euclidean distance or cosine distance between feature vectors as the similarity metric.

[0004] Pedestrian pictures contain main character information and useless background information. In order to enable the model to learn useful target information, convolutional neural networks or Transformers are usually used to extract features from pictures, and the extracted features are processed and learned accordingly, finally obtaining a recognition model with better recognition ability.

[0005] It is generally believed that in deep pyramid architecture networks, whether CNN or Transformer, different depths of network layers extract different levels of features: shallow feature maps represent low-level features, which are conducive to positioning; deep feature maps represent high-level features, which are conducive to classification. The current feature pyramid has obvious defects: it pays more attention to the scale of features rather than the level of features. For different individuals of the same type with complex and changeable appearance, it is necessary to extract features at multiple levels.

[0006] When learning features, we get the feature map of the last layer of the feature extraction backbone, which has a large number of channels but a small size. Therefore, if we only use this feature map to build a model, we may miss some detailed information in the image, such as the logo on the target person's clothes. Therefore, the robustness of single-level feature learning is not high enough, and the model often has poor adaptability when migrating data sets. Therefore, it is necessary to do multi-level feature learning on the model. Summary of the invention

[0007] In order to objectively and effectively combine the multi-level features of the backbone and fuse the feature information of different levels, the present invention proposes a pedestrian re-identification method based on twins multi-level features. The present invention extracts the feature maps of the third and fourth stages of twins-SVT, performs convolution, mean normalization, activation function, slicing, pooling and other operations on the feature maps of the third stage, and performs mean normalization, pooling and other operations on the feature maps of the fourth stage. Then, the two branches are jointly trained, and the models are trained using Cross-Entropy+Center loss and Triplet loss respectively. Finally, the trained model is used to achieve the pedestrian re-identification task.

[0008] The method of the present invention mainly comprises the following steps:

[0009] Step 1: Modify the native twins-SVT model by removing the final global average pooling (GAP) and fully connected classification layer. The output of the model is the output feature map of the fourth stage of the backbone network, denoted as f s4 ∈b×7×7×512; then extract the feature map of the third stage of the model to get f s3 ∈ b×14×14×256, where b represents the size of a batch.

[0010] Step 2: transform the feature map f s4 After a global average pooling, the redundant dimensions are compressed to obtain the feature vector f g ∈ b×512, and then f g After the fully connected layer, the classification prediction vector p is obtained g ∈b×num_classes, where num_classes represents the total number of classification categories in the data set;

[0011] Step 3: transform the feature map f s3 ∈b×14×14×256 passes through a Convblock module to obtain the feature map f′ with the same number of channels as the fourth stage of the backbone s3∈b×12×12×512.

[0012] The Convblock module includes a two-dimensional convolution, a BN layer, a SE attention, and a Mish activation function.

[0013] Step 4: f′ s3 Horizontally split, recorded as part-1, part-2, and the corresponding feature maps are recorded as get and

[0014] right and After global average pooling, the redundant dimensions are compressed to obtain the feature vector and Then and After the fully connected layer, the classification prediction vector is obtained and

[0015] Step 5: g , Input the triplet loss Triplet Loss and convert p g , Input cross-entropy loss + center loss for joint training; the definitions of the two loss functions are:

[0016]

[0017]

[0018]

[0019] In formula 1 represents the i-th picture of the a-th image, f(·) represents the features extracted by the model for the image, ||·|| represents the L2 norm, m is the margin value in the loss, N is the size of the batch, and the relationship between a, p, and n is: a and p are different pictures of the same person, and a and n are pictures of different people.

[0020] In formula 2, x i is the extracted feature of the i-th image in each batch, The yth i The class center of each category, B is the size of the batch size.

[0021] In formula 3, p iis the one-hot vector of the true label. A one-hot vector is a vector in which the position of the true target in the multi-classification label is recorded as 1 and the other positions are 0. i is the model’s predicted probability score for the i-th class.

[0022] When the training loss no longer shows a significant downward trend, the training optimization is completed and the model parameters are saved.

[0023] Step 6: Use the query set and gallery set in the dataset to evaluate the performance of the model. First, use the model to extract the representation vectors of all samples in the gallery set, where the representation vector is f g , The connection of the three vectors represents that each sample in the query set is extracted from its representation vector, and the Euclidean distance between the vectors in the gallery set is calculated. The closest samples are recorded as the query result set; finally, the accuracy of the query result is calculated to evaluate the quality of the model.

[0024] Step 7: Implement the actual recognition target application. Send all images of the data set to be queried into the model to obtain the representation vector table of all images (the representation vector of each image is f g , The image of the target person to be queried is then sent to the model to obtain the representation vector of the image. The distance to each vector in the vector table is calculated and sorted in ascending order of distance to obtain the top ranked images, which are the results of model re-identification.

[0025] Compared with many existing pedestrian re-identification methods, the present invention has the following characteristics:

[0026] First, for the feature extraction backbone, we chose twins-SVT, which has fewer parameters and a more novel architecture than the traditional ResNet50, and implemented it based on Transformer. Experiments have shown that the feature extraction capabilities of the two are comparable. In addition, we combined the feature maps between different layers of the backbone for joint training, which significantly helped improve the accuracy of the model.

[0027] Second, in the feature processing stage, the corresponding Convblock is designed for the feature map of the third stage, and SE attention and a smoother activation function Mish are added to the Convblock to make the number of channels of the feature maps of the two stages consistent. The feature map is also segmented so that the model can pay more attention to local information.

[0028] Third, in the supervised training stage, a BN layer is added between the feature vector and the predicted classification score, so that the problem of inconsistent optimization targets of different losses in later training is solved. The feature vector is fed into the triple loss, and the predicted classification score is fed into the cross entropy loss and center loss for joint training. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flow chart showing the pedestrian re-identification method of the present invention;

[0030] Figure 2 represents the twins-SVT backbone of the model;

[0031] Figure 3 It represents the visualization of the feature graph extracted by the feature extraction network of the embodiment for the pedestrian picture;

[0032] Figure 4 Represents a pedestrian re-identification model based on twins' multi-level features;

[0033] Figure 5 (a) and (b) respectively show the comparison of activation functions Mish and Relu;

[0034] 6(a), (b), and (c) are graphs showing the total loss of the model, the accuracy of the training set, and the change of the evaluation index during the training process of the embodiment, respectively;

[0035] Figure 7 The query result visualization of the embodiment. DETAILED DESCRIPTION

[0036] In order to effectively extract the features of pedestrians and perform classification and recognition, the present invention mainly improves the features extracted by twins-SVT. The following is a detailed description of the embodiments of the present invention in conjunction with the accompanying drawings: This embodiment is implemented based on the technical solution of the present invention, and a detailed implementation method and specific operation process are given.

[0037] The pedestrian re-identification method based on twins multi-level features has the following overall execution process: Figure 1 As shown, the specific implementation method includes the following steps:

[0038] Step 1: For the original twins-SVT model Figure 2 As shown in the figure, the model is modified to remove the final global average pooling (GAP) and fully connected classification layer. The output of the model is the output feature map of the fourth stage of the backbone network. The features of pedestrian images are extracted from the dataset Market-1501, denoted as f s4 ∈b×7×7×512; then extract the feature map of the third stage of the model to get f s3∈b×14×14×256, where b represents the size of a batch. The partial feature map extraction of the four stages of the backbone is visualized as follows: Figure 3 As shown, it can be seen that the feature map of the third stage retains more character detail features than the fourth stage.

[0039] Step 2, such as Figure 4 As shown in the overall architecture of the model, the feature map f s4 After a global average pooling, the redundant dimensions are compressed to obtain the feature vector f g ∈b×512, and then f g After the fully connected layer, the classification prediction vector p is obtained g ∈ b×num_classes, where num_classes represents the total number of classification categories in the data set;

[0040] Step 3: transform the feature map f s3 ∈b×14×14×256 passes through a Convblock module to obtain the feature map f′ with the same number of channels as the fourth stage of the backbone s3 ∈b×12×12×512. The Convblock module includes a two-dimensional convolution, a BN layer, SE attention, and a Mish activation function. In order to ensure the smoothness of gradient propagation during training, the traditional activation function Relu is changed to Mish. The comparison between the two is as follows: Figure 5 As shown, we can see that when x<0, the gradient of the Relu function remains unchanged at 1, while the gradient of the Mish function is in a gentle transformation range, which may be beneficial to the gradient descent update of the model parameters.

[0041] Then for f′ s3 Perform horizontal splitting to obtain and Again and After global average pooling, the redundant dimensions are compressed to obtain the feature vector and Then and After the fully connected layer, the classification prediction vector is obtained and

[0042] Step 4: f g , Input the triplet loss Triplet Loss and convert p g , The cross-entropy loss + center loss is sent in for joint training; the definitions of the three loss functions are:

[0043]

[0044]

[0045]

[0046] In formula 1 represents the i-th image of a, f(·) represents the features extracted by the model, ||·|| represents the L2 norm, m is the margin value in the loss, and the value of m is set to 0.3 during training. N is the batch size, which is set to 32, that is, 4 character IDs are input each time, and each character has 8 images. The relationship between a, p, and n is: a and p are different pictures of the same person, and a and n are pictures of different people.

[0047] In formula 2, x i is the extracted feature of the i-th image in each batch, The yth i The class center of each category, B is the size of the batch size.

[0048] In formula 3, p i is the one-hot vector of the true label. A one-hot vector is a vector in which the position of the true target in the multi-classification label is recorded as 1 and the other positions are 0. i is the model’s predicted probability score for the i-th class.

[0049] When the training loss no longer changes significantly, the training optimization is completed and the model parameters are saved. The changes in the loss, accuracy and evaluation indicators during the model training process are shown in Figure 6(a), (b) and (c), respectively.

[0050] Step 5: Use the query set and gallery set in the dataset to evaluate the performance of the model. First, use the model to extract the representation vectors of all samples in the gallery set (the representation vector is f g , The concatenation of three vectors is used to represent the query result. Then, the representation vector of each sample in the query set is extracted, and the Euclidean distance between the vectors is calculated with the vector table in the gallery. The closest samples are recorded as the query result set. Finally, the accuracy of the query result is calculated to evaluate the quality of the model. The mAP and rank-k of the model are shown in Table 1, where RK represents re-rank.

[0051] Table 1. mAP and rank-k scores of the models on market-1501

[0052]

[0053] Step 6: Implement the actual recognition target application. Send all the images of the data set to be queried into the model and obtain the representation vector table of all images (the representation vector of each image is f g , The image of the target person to be queried is then sent to the model to obtain the representation vector of the image. The distance to each vector in the vector table is calculated and sorted from small to large. The top images are obtained, which are the results of the model re-identification. The re-identification results and visualization are shown in Figure 2. Figure 7 shown.

Claims

1. A pedestrian re-identification method based on twins multi-level features, characterized in that: The construction of the model includes the following steps: Step 1: Use the twins-SVT transformer to extract different levels of features of pedestrian images from the Market-1501 dataset. The features extracted from stage 3 are recorded as feature maps f s3 , f s3 ∈14×14×256; the features extracted from stage4 are recorded as feature map f s4 , f s4 ∈7×7×512; Step 2: transform the feature map f s4 After a global average pooling, the redundant dimensions are compressed to obtain the feature vector f g , f g ∈b×512, and then f g After the fully connected layer, the classification prediction vector p is obtained g , p g ∈b×num_classes, where num_classes represents the total number of classification categories in the data set; Step 3: transform the feature map f s3 ∈b×14×14×256 passes through a Convblock module to obtain the feature map f′ with the same number of channels as the fourth stage of the backbone s3 ∈b×12×12×512; Step 4: f s3 The horizontal branch is divided into two branches, denoted as part-1 and part-2, and the corresponding feature maps are denoted as The size of these two feature maps is 6×12×512, and we get and The two feature maps are subjected to global average pooling and dimensionality reduction to obtain two feature vectors: After the BN layer and the fully connected layer, the classification prediction vectors are obtained respectively. and Step 5: g , Input the triplet loss Triplet Loss and convert p g , Send in Cross-Entropy Loss and Center Loss for joint training; When the training loss no longer shows a significant downward change, the training optimization is completed and the model parameters are saved; The definitions of the loss functions in step 5 are: In formula (1) represents the i-th image of the a-th image, f(·) represents the features extracted by the model, ‖·‖ represents the L2 norm, m is the margin value in the loss, N is the size of the batch, and the relationship between a, p, and n is: a and p are different images of the same person, and a and n are images of different people; In formula (2), x i is the extracted feature of the i-th image in each batch, The yth i The class center of each category, B is the size of the batch size; In formula (3), p i is the one-hot vector of the true label. A one-hot vector refers to a vector in which the position of the true target in the multi-classification label is recorded as 1 and the other positions are 0; q i is the model’s predicted probability score for the i-th class; Step 6: Use the query set and gallery set in the dataset to evaluate the performance of the model. First, use the model to extract the representation vectors of all samples in the gallery set, where the representation vector is f g , The concatenation of the three vectors extracts the representation vector of each sample in the query set, calculates the Euclidean distance between the vectors and the vector table in the gallery set, and records the closest samples as the query result set; finally, calculates the accuracy of the query result to evaluate the quality of the model; Step seven, realize the application of actual recognition target; send all images of the data set to be queried into the model, obtain the representation vector table of all images, and then send the image of the target person to be queried into the model to obtain the representation vector of the image, calculate the distance with each vector in the vector table, sort them from small to large by distance, obtain several images with the highest ranking, and then you can get the result of model re-recognition.

2. The pedestrian re-identification method based on twins multi-level features according to claim 1, characterized in that: The Convblock module includes: two-dimensional convolution, BN layer, SE attention layer, and Mish activation function.

Images