A dual-branch cross-domain person re-identification method based on attention calibration

Abstract

The present invention discloses a dual-branch cross-domain pedestrian re-identification method based on attention calibration, which uses an attention calibration module to improve the baseline network of ResNet-50, introduces a domain-invariant feature supplementary branch and a domain-specific feature global branch, linearly weights the domain-invariant feature information and the domain-specific feature information to obtain a fused feature, and normalizes the fused feature to obtain the final feature of the model. The baseline network of the present invention can fully extract the channel information and spatial information of the feature, and solve the problem of feature misalignment. By introducing a domain-specific feature global branch, the amount of model parameters can be reduced, and the training bias is corrected in combination with the baseline network, while the model complexity is reduced, so that the model can fully extract domain-specific features. By introducing a domain-invariant feature supplementary branch, the complementary information of the source domain is obtained from the calibration features of a specific field, and the domain-specific features extracted by the domain-specific feature global branch are combined, so that the model has a strong generalization ability.

Description

Technical Field

[0001] The present invention relates to a pedestrian re-identification method, and in particular to a dual-branch cross-domain pedestrian re-identification method based on attention calibration. Background Art

[0002] As one of the most attractive research tasks in the computer vision community, person re-identification aims to determine whether the same pedestrian is contained in non-overlapping camera views. It plays an important role in various surveillance applications, such as pedestrian retrieval and public safety event detection, and has therefore attracted the attention of many scholars in the past few years. However, after the pedestrian re-identification model is trained in the source domain, the performance often drops significantly when it is applied to the dataset of a new domain. Most of the existing research focuses on fully supervised person re-identification and has made encouraging progress. However, these methods are based on independent and identically distributed datasets. In order to improve this problem, scholars have introduced unsupervised domain adaptation. These works learn models from labeled source domains and unlabeled target domain data through image translation, feature alignment or multi-task learning, but such methods require a large amount of target domain data to work better. In addition to the unsupervised concept, scholars have proposed domain generalization technology for the real world. This technology uses multiple source domains for model training, and the model can perform well in unseen target domains without updating. After searching the existing technical literature, Choi et al. proposed the MetaBIN algorithm by combining batch normalization (BN) and instance normalization (IN) and used it to meta-train the batch instance normalization network, but this method requires very careful handling of the relationship between batch normalization and instance normalization. Song et al. used feature mapping and meta-learning to learn domain-invariant features to fully extract domain-invariant and domain-specific features. He et al. used knowledge distillation to train multi-source domain data through student models and teacher models, and transferred the knowledge learned by the teacher model to the student model with relatively weak learning ability to enhance the generalization ability of the student model. These methods are often too complicated and only eliminate style differences from a given source domain, so it lacks the ability to fully attenuate new styles in unseen domains. Wang et al. proposed a large-scale synthetic person re-identification dataset called RandPerson. The knowledge learned from synthetic data can also be well applied to real-world datasets. This method has alleviated the current lack of person re-identification datasets to a certain extent.

[0003] In summary, in cross-domain person re-identification, how to better improve the model's generalization ability for data sets is an urgent problem that technicians in this field need to solve. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a dual-branch cross-domain pedestrian re-identification method based on attention calibration.

[0005] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0006] The present invention is a dual-branch cross-domain pedestrian re-identification method based on attention calibration, comprising the following steps:

[0007] S1: Construct an end-to-end person re-identification network based on the attention calibration module;

[0008] S2: Generate feature maps using the person re-identification network, and create domain-invariant feature supplement branches and domain-specific feature global branches based on the feature maps to obtain a model framework for identifying and retrieving cross-domain pedestrian objects;

[0009] S3: Use the domain-invariant feature supplement branch to learn the domain-invariant feature information of the source domain dataset, and use the domain-specific feature global branch to learn the domain-specific feature information of the source domain dataset;

[0010] S4: Linearly weight the domain-invariant feature information and the domain-specific feature information to obtain fused features, and normalize the fused features to obtain the final features of the model;

[0011] S5: Based on the final features of the model, metric learning is used to calculate the similarity between features, and they are sorted by similarity. The top few features are selected as the model retrieval results.

[0012] A further improvement of the present invention is that the pedestrian re-identification network in S1 includes three convolution blocks and two attention calibration modules, and the original input features are sequentially passed through the two convolution blocks, the first attention calibration module, the convolution block and the second attention calibration module to obtain the output features.

[0013] A further improvement of the present invention is that the attention calibration module in S1 includes two convolutional layers, a feature calibration FCBN layer and a Sigmoid function, and the input feature f i After passing through the first convolution layer, the feature calibration FCBN layer, the second convolution layer and the Sigmoid layer in sequence, an attention-calibrated attention mask M is obtained, whose formula is defined as:

[0014] M=Sigmoid(Conv 2 (FCBN(Conv 1 (f i ))))

[0015] Among them, the feature calibration FCBN layer is expressed as:

[0016] FCBN(X (n,c,h,w) )=x (n.c,h,w) +ω·G

[0017] The input feature of the FCBN layer is the output feature x of the first convolutional layer, n represents the number of batches, c represents the number of channels, h and w represent the height and width of the input image, X represents the output feature of the FCBN layer, ω is a learnable weight vector, and G is the statistical information of feature X;

[0018] Add the feature map of the attention mask M to the input feature f i In order to emphasize the distinguishing features, we can get an attention-calibrated output feature F O , whose expression is:

[0019] F O =f i ·M+f i .

[0020] A further improvement of the present invention is that the domain-invariant feature supplement branch and the domain-specific feature global branch in S2 are respectively derived from the first attention calibration module and the second attention calibration module.

[0021] A further improvement of the present invention is that: the output feature X of the domain invariant feature supplement branch in S3 (DIFSB) It is expressed as follows:

[0022] X (DIFSB) =Conv a2 (IN(Relu(FCBN a2 (Conv a1 (FCBN a1 (x s ))))))

[0023] Among them, x s represents the features output from the first attention calibration module, IN is the style normalization layer, Conv a1 Represents the first convolutional layer of this branch, Conv a2 Represents the second convolutional layer of this branch, FCBN a1 、FCBN a2 Represents the first feature-calibrated FCBN layer and the second feature-calibrated FCBN layer of this branch.

[0024] A further improvement of the present invention is that the output feature X of the domain-specific feature global branch in S3 (DSFGB) It is expressed as follows:

[0025] X (DSFGB) =Conv b (FCBN b2 (Bottleneck(FCBN b1 (x t ))))

[0026] Among them, x t Represents the features output from the second attention calibration module, Conv b Represents the convolutional layer of this branch, which is used to reduce the number of feature channels c, FCBN b1 、FCBN b2 Represents the first feature-calibrated FCBN layer and the second feature-calibrated FCBN layer of this branch.

[0027] A further improvement of the present invention is that: S4 normalizes the fused features to obtain the output feature X:

[0028] X=Normalize(a×X (DSFGB) +b×X (DIFSB) )

[0029] Among them, a and b represent weight coefficients, and Normalize represents the L2 norm.

[0030] The beneficial effects of the present invention are: 1. The pedestrian re-identification network of the present invention is based on the ResNet-50 neural network framework, and combines the attention calibration module to improve the ResNet-50 baseline network. The baseline network can fully extract the channel information and spatial information of the features, solve the problem of feature misalignment and locate distinguishable local features, while reducing noise features to better extract feature information that is beneficial to pedestrian re-identification.

[0031] 2. The present invention introduces a domain-specific feature global branch, which includes two feature calibration FCBN layers and a Bottleneck layer. The feature calibration FCBN layer of the present invention can alleviate the training bias in the standard BN layer and improve the generalization performance. The Bottleneck layer can reduce the number of model parameters and can effectively train data and extract features. Combined with the baseline network, the training bias is corrected, so that some high-frequency noise is eliminated in the learned features, and the model complexity is reduced, so that the model can fully extract domain-specific features.

[0032] 3. The present invention introduces a domain-invariant supplementary branch and combines it with the benchmark network to enrich the complementary information of the source domain obtained from the calibration features of a specific field. It relies on the style normalization IN layer to filter out domain-specific information and obtain domain-invariant features. It combines the domain-specific features extracted by the global domain-specific branch, so that the model has a strong generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is the overall framework diagram of the present invention.

[0034] Figure 2 It is a schematic diagram of the experimental results of the method of the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0036] like Figure 1 The present invention is a dual-branch cross-domain pedestrian re-identification method based on attention calibration, and the specific steps are:

[0037] S1: Construct an end-to-end person re-identification network based on the attention calibration module; the specific operation is: select the first three convolution blocks of ResNet-50, and add the attention calibration module after the second convolution block and the third convolution block to form a baseline network. The attention calibration module includes two convolution layers, namely the first convolution layer and the second convolution layer, a feature calibration FCBN layer and a Sigmoid function, and the input feature f i It passes through the first convolution layer, the feature calibration FCBN layer, the second convolution layer and the Sigmoid activation function layer in sequence to improve the nonlinear expression ability of the model and obtain an attention-calibrated attention mask M. The attention mask M is expressed as:

[0038] M=Sigmoid(Conv 2 (FCBN(Conv 1 (f i ))))

[0039] Both Conv are 1×1 convolutions. 1 It is the first convolutional layer of the attention calibration module, which is used to reduce the dimension of features. 2 It is the second convolutional layer of the attention calibration module, which is used to increase the dimension of features.

[0040] The feature calibration FCBN layer is improved from the standard BN layer, and its formula is as follows:

[0041] FCBN(X 1(n,c,h,w) )=x (n.c,h,w) +ω·G

[0042] Among them, x (n,c,h,w) represents the input feature x, is the output feature of the first convolutional layer, n represents the number of batches, c represents the number of channels, h and w represent the height and width of the input image; where ω is a learnable weight vector, ω∈R 1×C×1×1 , R represents a real number, G is the feature X 1 The statistics of can have various shapes, such as G∈R N×1×H×W Or G∈R N×1×H×W In this embodiment, G is set by default to the average value μ(μ∈R N×C×1×1 ). First, broadcast the two features to the same shape and then perform a dot product operation. · represents the dot product operation. The output feature X is obtained by FCBN.1 .

[0043] The feature map of the attention mask M is added to the input feature f i In order to emphasize the distinguishing features, we can get an attention-calibrated output feature F O , the formula is as follows:

[0044] F O =f i ·M+f i

[0045] Here, the operator · represents the dot product operation.

[0046] S2: The domain-invariant feature supplementation branch (DIFSB) is derived from the first attention calibration module of the constructed baseline network, and the domain-specific feature global branch (DSFGB) is derived from the second attention calibration module.

[0047] The domain-invariant feature supplementation branch includes two feature calibration FCBN layers, two convolutional layers, ReLU function, and style normalization IN layer. The input features of this branch enter the convolutional layer after passing through the first feature calibration FCBN layer, and then pass through the second feature calibration FCBN layer, ReLU activation function, style normalization IN layer, and convolutional layer to output domain-invariant features.

[0048] The domain-specific feature global branch includes two feature calibration FCBN layers, a Bottleneck layer and a convolutional layer; the input features of this branch pass through the first feature calibration FCBN layer, enter the Bottleneck layer, then pass through the second feature calibration FCBN layer, and finally output the domain-specific features after the convolutional layer.

[0049] The output feature X of the domain invariant feature supplementation branch (DIFSB) (DIFSB) The formula is as follows:

[0050] X (DIFSB) =Conv a2 (IN(Relu(FCBN a2 (Conv a1 (FCBN a1 (x s ))))))

[0051] Among them, x s represents the features output from the first attention calibration module, IN is the style normalization layer, Conv is the convolutional layer, Conv a1 Represents the first convolutional layer of this branch, which is used to reduce the height h and width w of the image. Conv a2 Represents the second convolutional layer of this branch, which is used to reduce the number of feature channels c, FCBN a1 、FCBNa2 Represents the first feature-calibrated FCBN layer and the second feature-calibrated FCBN layer of this branch.

[0052] Domain-specific features Output features X of the global branch (DSFGB) The formula is as follows:

[0053] X (DSFGB) =Conv b (FCBN b2 (Bottleneck(FCBN b1 (x t ))))

[0054] Among them, x t Represents the features output from the second attention calibration module. The convolutional layer Conv b , used to reduce the number of feature channels c, FCBN b1 、FCBN b2 Represents the first feature-calibrated FCBN layer and the second feature-calibrated FCBN layer of this branch.

[0055] The style normalization IN layer normalizes the features in the following way:

[0056]

[0057] where x in Represents the input features from the ReLU activation function, ready to enter the IN layer, X in Represents its output features. μ∈R c and δ∈R c are the mean and standard deviation calculated for the mini-batch, γ∈R c and β∈R c are affine parameters, and ε>0 is to avoid division by a constant zero. During training, μ and δ are estimated by a moving average operation.

[0058] S4: Linearly weight the two feature information learned by the domain invariant feature supplement branch (DIFSB) and the domain specific feature global branch (DSFGB) to obtain the fused feature, and then use the L2 norm to standardize the fused feature to obtain the final feature X of the model:

[0059] X=Normalize(a×X (DSFGB) +b×X (DIFSB) )

[0060] Where X (DSFGB) represents the output features of the global branch of domain-specific features, X (DIFSB)represents the output feature of the domain invariant feature supplement branch, a and b represent weight coefficients, which are set to 0.9 and 0.1 respectively in this embodiment, and Normalize represents the L2 norm;

[0061] S5: Use metric learning to calculate the similarity between the final features, sort them by similarity, and take the top ten images as the results retrieved by the model. The calculation can be performed using a contrast loss function or a triplet loss function.

[0062] In this implementation, batch OHEM triplet loss is used to calculate the similarity between features, and its OHEM triplet loss function L tri The formula is as follows:

[0063]

[0064] in From small batch samples, represents the jth picture of the i-th person. The hardest negative represents people who look similar but are not the same person, and the hardest positive represents people who look different but are the same person. Δ represents the network parameter, and f Δ is the feature extractor, m is the margin, and D(·,·) represents the similarity between features.

[0065] In this example, the training batch size is set to 64 and the test batch size is set to 128. We use the classic SGD optimizer to train the model, and the learning rate of the backbone is 0.0005. Figure 2 As shown in the figure, the first and fourth columns are pictures input to the network, the second and fifth columns are visualized activation features after the first attention calibration module, and the third and sixth columns are visualization results of the activation features output by the second attention calibration module. It can be seen from the figure that the method of the present invention can better focus on the identifiable features of pedestrians, such as hairstyles, bags, and corners of clothes.

Claims

1. A dual-branch cross-domain person re-identification method based on attention calibration, characterized by: The steps include: S1: Construct an end-to-end person re-identification network based on the attention calibration module; The person re-identification network in S1 consists of three convolution blocks and two attention calibration modules. The original input features are sequentially passed through two convolution blocks, the first attention calibration module, the convolution block, and the second attention calibration module to obtain the output features; The attention calibration module in S1 consists of two convolutional layers, a feature calibration FCBN layer, and a Sigmoid function. The input feature f i After passing through the first convolution layer, the feature calibration FCBN layer, the second convolution layer and the Sigmoid layer in sequence, an attention-calibrated attention mask M is obtained, whose formula is defined as: M=Sigmoid(Conv2(FCBN(Conv1(fi)))); Among them, the feature calibration FCBN layer is expressed as: FCBN(X (n,c,h,w) )=x (n.c,h,w) +ω·G; The input feature of the FCBN layer is the output feature x of the first convolutional layer, n represents the number of batches, c represents the number of channels, h and w represent the height and width of the input image, X represents the output feature of the FCBN layer, ω is a learnable weight vector, and G is the statistical information of feature X; Add the feature map of the attention mask M to the input feature f i In order to emphasize the distinguishing features, we can get an attention-calibrated output feature F O , whose expression is: F O =f i ·M+f i ; S2: Generate feature maps using the person re-identification network, and create domain-invariant feature supplement branches and domain-specific feature global branches based on the feature maps to obtain a model framework for identifying and retrieving cross-domain pedestrian objects; S3: Use the domain-invariant feature supplement branch to learn the domain-invariant feature information of the source domain dataset, and use the domain-specific feature global branch to learn the domain-specific feature information of the source domain dataset; S4: Linearly weight the domain-invariant feature information and the domain-specific feature information to obtain fused features, and normalize the fused features to obtain the final features of the model; S5: Based on the final features of the model, metric learning is used to calculate the similarity between features and sort them by similarity.

2. According to claim 1, a dual-branch cross-domain pedestrian re-identification method based on attention calibration is characterized by: The domain-invariant feature supplementation branch and the domain-specific feature global branch in S2 are led by the first attention calibration module and the second attention calibration module, respectively.

3. According to claim 1, a dual-branch cross-domain pedestrian re-identification method based on attention calibration is characterized by: The output feature X of the domain invariant feature supplementation branch in S3 (DIFSB) It is expressed as follows: Among them, x s represents the features output from the first attention calibration module, IN is the style normalization layer, Conv a1 Represents the first convolutional layer of this branch, Conv a2 Represents the second convolutional layer of this branch, FCBN a1 、FCBN a2 Represents the first feature-calibrated FCBN layer and the second feature-calibrated FCBN layer of this branch.

4. According to claim 1, a dual-branch cross-domain pedestrian re-identification method based on attention calibration is characterized by: The output feature X of the global branch of domain-specific features in S3 (DSFGB) It is expressed as follows: X (DSFGB) =Conv b (FCBN b2 (Bottleneck(FCBN b1 (x t )))); Among them, x t Represents the features output from the second attention calibration module, Conv b Represents the convolutional layer of this branch, which is used to reduce the number of feature channels c, FCBN a1 、FCBN a2 Represents the first feature-calibrated FCBN layer and the second feature-calibrated FCBN layer of this branch.

5. According to claim 1, a dual-branch cross-domain pedestrian re-identification method based on attention calibration is characterized by: S4 normalizes the fusion features and obtains the output feature X: X=Normalize(a×X (DSFGB) +b×X (DIFSB) ); Among them, a and b represent weight coefficients, and Normalize represents the L2 norm.

Images