Cross-domain person re-identification method based on attention guidance and multi-scale label generation

Abstract

The present invention discloses a cross-domain pedestrian re-identification method based on attention guidance and multi-scale label generation. The present invention proposes a progressive learning framework based on attention guidance and multi-scale label generation. First, a background constraint module guided by an attention mechanism is designed to highlight the useful foreground pedestrian information in the image and reduce the interference of background clutter. This is conducive to guiding the model to learn more discriminative features related to the identity of pedestrians, thereby improving the recognition of pedestrian features. Secondly, in order to mine the detailed feature information in the image, the model mines the potential similarities from global to local, from shallow to deep, and generates pseudo labels for pedestrians from different perspectives. The results show that this method has better robustness and adaptability, and effectively enhances the generalization ability of the model.

Description

Technical Field

[0001] The invention of this article relates to the technical field of cross-domain pedestrian re-identification, and specifically to a cross-domain pedestrian re-identification method based on attention guidance and multi-scale label generation. Background Art

[0002] Person re-identification aims to match images of the same pedestrian from different cameras given a pedestrian image. It is widely used in the field of intelligent surveillance and is an important topic in computer vision. However, most of the proposed methods are limited to a single scene dataset and a large amount of annotated data. If the model trained using the dataset is directly deployed in a new scene, the performance of the model will often drop significantly. At the same time, it is expensive to build a dataset for each scene, which greatly limits the practical application of pedestrian re-identification technology.

[0003] Therefore, in order to solve this kind of problem, an unsupervised domain adaptation method is proposed. By using annotated source domain datasets and unlabeled target domain datasets, the performance of the model trained in the source domain can be improved in the target domain. There are two existing solutions, namely, the Gan-based style transfer method and the clustering-based pseudo-label annotation method. The Gan-based style transfer method converts the style of the source domain image into the style of the target domain image by using the Gan network, and uses the labeled source domain data after style transfer for feature learning, which narrows the field gap to a certain extent. However, compared with the clustering-based pseudo-label annotation method, the performance of the former is relatively weak, and the pseudo-label annotation method is currently widely used. This method first uses the model trained in the source domain to extract the features of the target domain image, and applies the unsupervised clustering method to predict the pseudo-label in the target domain, so as to train the network using the target domain samples annotated with pseudo-labels, and gradually improves the performance of the re-ID model in the target domain by alternating pseudo-label generation and fine-tuning the network.

[0004] During the research and development process, the inventors of the present invention found that in the clustering-based cross-domain pedestrian re-identification method, the entire image is usually used as a sample for training, ignoring a problem that the image background will cause deviations. The features extracted by the model include pedestrian features and background features. Therefore, in the clustering process, due to the similar backgrounds of pedestrians captured by the same camera, the pedestrian features extracted by the same camera tend to be clustered into the same category. In addition, the clustering process uses a single-scale global feature for pseudo-labeling, ignoring the potential similarity of unlabeled samples at different stages of the backbone network, and also ignoring the similarity between local features of pedestrian images. Summary of the invention

[0005] The purpose of the present invention is to provide a cross-domain pedestrian re-identification method based on attention guidance and multi-scale label generation to address the shortcomings of the prior art.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] Step (1) introduce the attention mechanism into the residual network ResNet50, add multi-scale branches, and build a custom neural network;

[0008] Step (2) uses the labeled source domain data to train the neural network constructed in step (1) to obtain an initial re-ID model;

[0009] Step (3) inputting the source domain image into the StarGan model for style transfer, so that the source domain image acquires the style of the corresponding camera in the target domain, and obtains the source domain dataset after style transfer, and uses the dataset to fine-tune the initial re-ID model;

[0010] Step (4) inputting the unlabeled target domain data into the re-ID model obtained in step (3) to extract features and obtain multi-scale feature grouping for each image;

[0011] Step (5) according to the multi-scale feature grouping generated in step (4), respectively performing distance calculation to construct a distance matrix, clustering each group in combination with a clustering algorithm, and annotating pseudo labels for the target domain image according to the clustering structure;

[0012] Step (6) uses the target domain data annotated with multi-scale pseudo labels and a progressive training strategy to train the re-ID model obtained in step (3);

[0013] Step (7) Iterate steps (4) to (6) until the model is stable; use the final re-ID model to retrieve the query image in the test set to obtain the final result.

[0014] Furthermore, the specific implementation process of step (1) is as follows:

[0015] 1-1 uses ResNet50 pre-trained on ImageNet as the backbone network, retains the four stages of the backbone, removes the last spatial downsampling operation, global average pooling layer and fully connected layer, and adds an instance normalization layer to the first BN layer in the residual block;

[0016] 1-2 Insert an attention-guided background constraint module between each stage of ResNet50. The background constraint module includes a forward attention mechanism and a reverse attention mechanism.

[0017] The forward attention mechanism consists of two parts: channel attention C and spatial attention S, which can be written as the tensor product of S and C, as shown in formula (1);

[0018] ATT=Sigmoid(S×C) (1)

[0019] The reverse attention mechanism is formed by the features abandoned by the forward attention mechanism, solving the problem of feature information loss caused by the forward attention mechanism. The kernel of the reverse attention mechanism is shown in formula (2);

[0020] ATT R =1-Sigmoid(S×C) (2)

[0021] 1-3 A multi-scale branch that extracts features from stages 2 and 3 of ResNet50, that is, extracts the shallow and middle-level features generated by the backbone network from stages 2 and 3 respectively; adds a fully connected layer and a local branch with a slicing operation after stage 4 of ResNet50, and obtains the global features, upper half features, and lower half features of pedestrians through the local branch; and adds a BN layer and a classification linear layer at the end, and the output is used for cross-entropy loss training network.

[0022] Furthermore, the specific implementation process of step (2) is as follows:

[0023] 2-1 Source domain pre-training combines difficult triplet loss and Softmax cross entropy loss as the loss function, where the output features of the first fully connected layer after the 4th stage of ResNet50 are used for difficult triplet loss, and the output of the classification linear layer is used for cross entropy loss; use this loss function to train until the model converges to obtain the initial re-ID model trained with source domain data.

[0024] Furthermore, the specific implementation process of step (3) is as follows:

[0025] 3-1 In order to learn an effective style transfer model, a camera style transfer model based on StarGAN is constructed. Assuming that there are C cameras in the target domain set, a StarGAN model is first trained, which can realize image-to-image conversion between each camera pair;

[0026] 3-2 For each image X in the source domain s,i , use the StarGAN model to generate C images, each image contains image X s,i The same pedestrian in the source domain, that is, the pedestrian label of the generated image is consistent with the original image, and the image style corresponds to the corresponding camera style in the target domain; the generated style transfer samples are combined with the original samples in the source domain to form a new training set, and the initial model obtained in step (2) is fine-tuned using the difficult triplet loss.

[0027] Furthermore, the specific implementation process of step (4) is as follows:

[0028] 4-1 For each image in the target domain dataset Use the re-ID model to extract features; the target domain image Input the re-ID model to obtain the pedestrian global features, upper half features and lower half features, as well as the shallow features and middle features generated in the backbone network; and these multi-scale features are subjected to the GAP operation; therefore, the target domain image The input model can extract five sets of feature vectors, namely global features Upper part features Lower part features Shallow Features and mid-level features

[0029] Furthermore, the specific implementation process of step (5) is as follows:

[0030] 5-1 The generated multi-scale feature groups are grouped to calculate the similarity between each other, and the K-reciprocal coding method is introduced to obtain the pedestrian feature distance matrix. Then, the unsupervised clustering method DBSCAN is used to cluster the multi-scale feature vector groups respectively, and the samples belonging to the same cluster are assigned the same pseudo label. A new target data set is established based on the grouping results of the 5 feature vectors. Each image has 5 pseudo labels, as shown in formula (3);

[0031]

[0032] in, N t Indicates the number of target domain images.

[0033] Furthermore, the specific implementation process of step (6) is as follows:

[0034] 6-1 Use the target domain data with multi-scale pseudo labels as supervision information, and use the difficult triplet loss to train the network for each pseudo label group of each scale. The difficult triplet loss randomly samples P identities, extracts K instances from each identity, and forms a mini batch of size P*K; select each image x in the batch in turn a As an anchor point, select the positive sample image x with the longest distance in the batch p and the nearest negative sample image x n The network is trained by forming triplets to enhance the generalization ability of the network; the formula is:

[0035]

[0036] 6-2 When the network reaches the best performance using the difficult triplet loss, the cross entropy loss is added to further optimize the network; thereby solving the error amplification problem caused by the use of cross entropy loss in the early stage of training, and also avoiding the model from falling into the local optimal solution problem using triplet loss. The formula is as follows:

[0037]

[0038] Among them, C1 represents the number of pedestrian categories in the target domain;

[0039] 6-3 When the model is stable, the final re-ID model is obtained. The query image and the test set image are input into the final re-ID model for feature extraction. The cosine distance between the query image features and the test set image features is compared, and the most matching pedestrian image is output.

[0040] The beneficial effects of the present invention are as follows:

[0041] The present invention designs a background constraint module guided by an attention mechanism, aiming to highlight useful foreground pedestrian information in the image and reduce the interference of background clutter. This is conducive to guiding the model to learn more discriminative features related to pedestrian identity, thereby improving the recognition of pedestrian features. Secondly, in order to mine the detailed feature information in the image, the model mines potential similarities from global to local, from shallow to deep, and generates pseudo labels for pedestrians from different perspectives. The results show that this method has better robustness and adaptability, and effectively enhances the generalization ability of the model. Compared with the most advanced cross-domain pedestrian re-identification method SNR, the present invention has a significant performance improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the overall implementation scheme of the present invention;

[0043] Figure 2 is a detailed schematic diagram of the background constraint module of the present invention;

[0044] Figure 3 is a schematic diagram of multi-scale feature extraction of the present invention;

[0045] Figure 4 Schematic diagram of Star-GAN image enhancement of the present invention;

[0046] Figure 5 It is a schematic diagram of multi-scale feature grouping and clustering of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described below in conjunction with the accompanying drawings.

[0048] Reference Figure 1 This is a flow chart of the overall implementation scheme of the present invention. The present invention reduces the inter-domain differences caused by background bias between different data sets by adding an attention-guided background constraint module, generates pseudo labels for pedestrians from different perspectives using multi-scale features, builds a multi-scale pseudo label group for each pedestrian, and mines potential similarities from global to local, from shallow to deep layers, effectively enhancing the generalization ability of the model. The present invention includes the following steps:

[0049] Step (1) introduce the attention mechanism into the residual network ResNet50 commonly used in the field of pedestrian re-identification, add multi-scale branches, and build a custom neural network;

[0050] Step (2) uses the labeled source domain data to train the neural network constructed in step (1) to obtain an initial re-ID model;

[0051] Step (3) inputting the source domain image into the StarGan model for style transfer, so that the source domain image acquires the style of the corresponding camera in the target domain, and obtains the source domain dataset after style transfer, and uses the dataset to fine-tune the initial re-ID model;

[0052] Step (4) inputting the unlabeled target domain data into the re-ID model obtained in step (3) to extract features and obtain multi-scale feature grouping for each image;

[0053] Step (5) according to the multi-scale feature grouping generated in step (4), respectively performing distance calculation to construct a distance matrix, clustering each group in combination with a clustering algorithm, and annotating pseudo labels for the target domain image according to the clustering structure;

[0054] Step (6) uses the target domain data annotated with multi-scale pseudo labels and a progressive training strategy to train the re-ID model obtained in step (3);

[0055] Step (7) Iterate steps (4) to (6) until the model is stable. Use the final re-ID model to retrieve the query image in the test set to obtain the final result.

[0056] The specific implementation process of step (1) is as follows:

[0057] 1-1 uses ResNet50 pre-trained on ImageNet as the backbone network, retains the four stages of the backbone, removes the last spatial downsampling operation, global average pooling layer and fully connected layer, and adds instance normalization in the first BN layer in the residual block.

[0058] 1-2 Insert the attention-guided background constraint module between each stage of ResNet50, refer to Figure 2 The attention adopted consists of two parts: channel attention C and spatial attention S, which can be written as the tensor product of S and C, as shown in formula (1).

[0059] ATT=Sigmoid(S×C) (1)

[0060] The features abandoned by the attention mechanism are used to form an anti-attention branch to solve the problem of feature information loss caused by the attention module. The anti-attention kernel is shown in formula (2).

[0061] ATT R =1-Sigmoid(S×C) (2)

[0062] 1-3 Extract the multi-scale branch of features from ResNet50 stages 2 and 3, that is, extract the shallow and middle-level features generated in the backbone network. Add a fully connected layer and a local branch with a slice operation after ResNet50 stage 4 to obtain the three-scale features of the pedestrian's full body, upper body, and lower body. Figure 3 . And add a BN layer and a classification linear layer at the end, and the output is used to train the network with cross entropy loss.

[0063] The specific implementation process of step (2) is as follows:

[0064] 2-1 Source domain pre-training combines difficult triplet loss and Softmax cross entropy loss as loss functions, where the output features of the first fully connected layer after the fourth stage of ResNet50 are used for difficult triplet loss, and the output of the classification linear layer is used for cross entropy loss. Use this loss function to train until the model converges, and obtain the initial re-ID model trained with source domain data.

[0065] The specific implementation process of step (3) is as follows:

[0066] 3-1 In order to learn an effective style transfer model, we build a camera style transfer model based on StarGAN. Assuming that we have C cameras in the target domain set, we first train a StarGAN model that can achieve image-to-image conversion between each camera pair.

[0067] 3-2 For each image X in the source domain s,i , use the StarGAN model to generate C images, each image contains image X s,i The same pedestrian in the target domain, that is, the pedestrian label of the generated image is consistent with the original image, and the image style corresponds to the corresponding camera style in the target domain, refer to Figure 4 After style transfer, the number of source domain images is expanded to C times the original number, and includes images with similar styles to the target domain. For example, in the Market-1501 dataset, the number of training set images is expanded from 12936 to 116424. The generated style transfer samples are combined with the original source domain samples to form a new training set, and the initial model obtained in step (2) is fine-tuned using the triplet loss.

[0068] The specific implementation process of step (4) is as follows:

[0069] 4-1 For each image in the target domain dataset Use the re-ID model to extract features. We take the target domain image Input the re-ID model to obtain the pedestrian global features, upper half features and lower half features, as well as the shallow and middle layer features generated in the backbone network. These multi-scale features are then subjected to the GAP operation. The input model can extract five sets of feature vectors, namely overall features Upper part features Lower part features Shallow Features and mid-level features 6. According to the cross-domain person re-identification method based on attention guidance and multi-scale label generation according to claim 5, it is characterized in that the specific implementation process of step (5) is as follows:

[0070] 5-1 The generated multi-scale feature vectors are grouped and their similarities are calculated. The K-reciprocal coding method is introduced to obtain the pedestrian feature distance matrix. Then, the unsupervised clustering method DBSCAN is used to cluster the multi-scale feature vector groups respectively, and the samples belonging to the same cluster are assigned the same pseudo label. Therefore, each image obtains 5 pseudo labels of different scales. Figure 5 We build a new target dataset based on the grouping results of the five feature vectors, and each image has five pseudo labels, as shown in formula (3).

[0071]

[0072]

[0073] The specific implementation process of step (6) is as follows:

[0074] 6-1 Use the target domain data with multi-scale pseudo labels as supervision information, and use the difficult triplet loss to train the network for each pseudo label group of each scale. The difficult triplet loss randomly samples P identities, extracts K instances from each identity, and forms a mini batch of size P*K. Select each image x in the batch in turn. a As an anchor point, select the positive sample image x with the longest distance in the batch p and the nearest negative sample image x n The three-tuple is formed to train the network and enhance the generalization ability of the network. The formula is:

[0075]

[0076] 6-2 When the network reaches the best performance using the difficult triplet loss, we add the cross entropy loss to further optimize the network. This strategy solves the error amplification problem caused by the use of cross entropy loss in the early stage of training, and also avoids the model falling into the local optimal solution problem using triplet loss. The formula is as follows:

[0077]

[0078] 6-3 When the model is stable, the final re-ID model is obtained, and the query image and the test set image are input into the final re-ID model for feature extraction. The cosine distance between the query image feature and the test set image feature is compared, and the most matching pedestrian image is output. As shown in Table 1, compared with other existing methods, the experimental results of the present invention on the Market-1501 and DukeMTMC-reID datasets show that our method has a mAP of 60.2% and a Rank-1 of 77.5% in the M→D experiment; in the D→M experiment, the mAP reached 70.2% and the Rank-1 reached 85.7%. Compared with the most advanced cross-domain pedestrian re-identification method, SNR has a significant performance improvement.

[0079] Table 1 Comparison results between the present invention and other existing methods

[0080]

Claims

1. The cross-domain person re-identification method based on attention guidance and multi-scale label generation is carried out in the following steps: Step (1) introduce the attention mechanism into the residual network ResNet50, add multi-scale branches, and build a custom neural network; Step (2) uses the labeled source domain data to train the neural network constructed in step (1) to obtain an initial re-ID model; Step (3) inputting the source domain image into the StarGan model for style transfer, so that the source domain image acquires the style of the corresponding camera in the target domain, and obtains the source domain dataset after style transfer, and uses the dataset to fine-tune the initial re-ID model; Step (4) inputting the unlabeled target domain data into the re-ID model obtained in step (3) to extract features and obtain multi-scale feature grouping for each image; Step (5) according to the multi-scale feature grouping generated in step (4), respectively performing distance calculation to construct a distance matrix, clustering each group in combination with a clustering algorithm, and annotating pseudo labels for the target domain image according to the clustering structure; Step (6) uses the target domain data annotated with multi-scale pseudo labels and a progressive training strategy to train the re-ID model obtained in step (3); Step (7) Iterate step (4) to step (6) until the model is stable; use the final re-ID model to retrieve the query image in the test set to obtain the final result; The specific implementation process of step (1) is as follows: 1-1 uses ResNet50 pre-trained on ImageNet as the backbone network, retains the four stages of the backbone, removes the last spatial downsampling operation, global average pooling layer and fully connected layer, and adds an instance normalization layer to the first BN layer in the residual block; 1-2 Insert an attention-guided background constraint module between each stage of ResNet50. The background constraint module includes a forward attention mechanism and a reverse attention mechanism. The forward attention mechanism consists of two parts: channel attention C and spatial attention S, which can be written as the tensor product of S and C, as shown in formula (1); ATT=Sigmoid(S×C) (1) The reverse attention mechanism is formed by the features abandoned by the forward attention mechanism, solving the problem of feature information loss caused by the forward attention mechanism. The kernel of the reverse attention mechanism is shown in formula (2); TO R =1-Sigmoid(S×C) (2) 1-3 extracts the multi-scale branch of features from stages 2 and 3 of ResNet50, that is, extracts the shallow and middle-level features generated by the backbone network from stages 2 and 3 respectively; A fully connected layer and a local branch with a slicing operation are added after the fourth stage of ResNet50 to obtain the global features, upper half features, and lower half features of pedestrians through the local branches; And add a BN layer and a classification linear layer at the end, whose output is used to train the network with cross entropy loss; The specific implementation process of step (5) is as follows: 5-1 The generated multi-scale feature groups are grouped to calculate the similarity between each other, and the K-reciprocal coding method is introduced to obtain the pedestrian feature distance matrix. Then, the unsupervised clustering method DBSCAN is used to cluster the multi-scale feature vector groups respectively, and the samples belonging to the same cluster are assigned the same pseudo label. A new target data set is established based on the grouping results of the 5 feature vectors. Each image has 5 pseudo labels, as shown in formula (3); in, N t Indicates the number of target domain images.

2. The cross-domain person re-identification method based on attention guidance and multi-scale label generation according to claim 1 is characterized in that The specific implementation process of step (2) is as follows: 2-1 Source domain pre-training combines difficult triplet loss and Softmax cross entropy loss as the loss function, where the output features of the first fully connected layer after the 4th stage of ResNet50 are used for difficult triplet loss, and the output of the classification linear layer is used for cross entropy loss; use this loss function to train until the model converges to obtain the initial re-ID model trained with source domain data.

3. The cross-domain person re-identification method based on attention guidance and multi-scale label generation according to claim 2 is characterized in that The specific implementation process of step (3) is as follows: 3-1 In order to learn an effective style transfer model, a camera style transfer model based on StarGAN is constructed. Assuming that there are C cameras in the target domain set, a StarGAN model is first trained, which can realize image-to-image conversion between each camera pair; 3-2 For each image X in the source domain s,i , use the StarGAN model to generate C images, each image contains image X s,i The same pedestrian in the source domain, that is, the pedestrian label of the generated image is consistent with the original image, and the image style corresponds to the corresponding camera style in the target domain; the generated style transfer samples are combined with the original samples in the source domain to form a new training set, and the initial model obtained in step (2) is fine-tuned using the difficult triplet loss.

4. The cross-domain person re-identification method based on attention guidance and multi-scale label generation according to claim 3 is characterized in that The specific implementation process of step (4) is as follows: 4-1 For each image in the target domain dataset Extract features using the re-ID model; The target domain image Input the re-ID model to obtain the pedestrian's global features, upper half features, lower half features, and shallow layer features and middle layer features generated in the backbone network; These multi-scale features are subjected to GAP operation; therefore, the target domain image The input model can extract five sets of feature vectors, namely global features Upper part features Lower part features Shallow Features and mid-level features 5. The cross-domain person re-identification method based on attention guidance and multi-scale label generation according to claim 4 is characterized in that The specific implementation process of step (6) is as follows: 6-1 Use the target domain data with multi-scale pseudo labels as supervision information, and use the difficult triplet loss to train the network for each pseudo label group of each scale. The difficult triplet loss randomly samples P identities, extracts K instances from each identity, and forms a mini batch of size P*K; select each image x in the batch in turn a As an anchor point, select the positive sample image x with the longest distance in the batch p and the nearest negative sample image x n Form triplets to train the network and enhance the generalization ability of the network; The formula is: 6-2 When the network reaches the best performance using the difficult triplet loss, the cross entropy loss is added to further optimize the network; thereby solving the error amplification problem caused by the use of cross entropy loss in the early stage of training, and also avoiding the model from falling into the local optimal solution problem using triplet loss. The formula is as follows: Among them, C1 represents the number of pedestrian categories in the target domain; 6-3 When the model is stable, the final re-ID model is obtained. The query image and the test set image are input into the final re-ID model for feature extraction. The cosine distance between the query image features and the test set image features is compared, and the most matching pedestrian image is output.

Images