A cross-domain pedestrian re-identification method based on point-by-point feature matching

Abstract

The application provides a cross-domain pedestrian re-identification method based on point-by-point feature matching. Firstly, in order to solve the problem that the network has low robustness to the cross-domain pedestrian image style, color and the like, an instance normalization layer is introduced into a Resnet50 basic network to extract image features and improve the generalization ability of the model. Secondly, in order to better capture local and global information and enhance the representation ability of features, a Transformer self-attention module is combined with convolution. Finally, a point-by-point feature mapping relationship is established in deep features, the image matching is regarded as a process of finding a local optimum point by point, the image matching similarity is calculated, and loss constraint is performed by using Focal loss. The application is mainly applied to a video monitoring intelligent analysis application system and has broad application prospects in the fields of image retrieval and intelligent security.

Description

Technical Field

[0001] This invention relates to a cross-domain pedestrian re-identification method based on point-by-point feature matching, belonging to the field of computer vision and intelligent information processing. Background Technology

[0002] Pedestrian re-identification is a popular research topic in computer vision, aiming to solve the problem of matching images of the same person across different cameras. Pedestrian re-identification has wide applications in intelligent video surveillance and smart security, such as searching for lost children and elderly people, investigating illegal and criminal activities, and tracking suspects. However, in real-world scenarios, due to differences in camera positions and angles, pedestrian postures, and occlusion, many methods exhibit poor generalization ability in new and unknown scenarios, posing a significant challenge to pedestrian re-identification tasks.

[0003] To address this issue, researchers have proposed unsupervised domain adaptation methods, primarily including clustering-based pseudo-labeling and style transfer-based methods. These methods largely compute a fixed representation vector for each image and use typical distance or similarity metrics for image matching, without considering the actual content correspondence between the two input images. Instead, they process each image independently, like classification, resulting in a model that extracts fixed feature representations. Without domain adaptation and transfer learning, the learned model is invariant, exhibiting poor generalization ability and unsuitability for handling unfamiliar scenarios. Compared to traditional cross-dataset approaches, unsupervised domain adaptation methods offer performance improvements. However, these methods require further learning in the target domain, consuming significant resources and time during deployment, limiting their application in real-world scenarios. Therefore, improving the generalization ability of baseline models is crucial. Summary of the Invention

[0004] This invention proposes a cross-domain person re-identification method based on point-by-point feature matching. An IN layer is introduced into ResNet50 as the backbone network to enhance the network model's domain adaptability. To better extract useful pedestrian features, a Transformer self-attention mechanism is introduced, utilizing its multi-head attention mechanism to extract better local and global pedestrian features. Finding local optima in different domains has universal applicability and can be well applied to unknown scenarios. Therefore, the image matching problem is viewed as a process of point-by-point matching to find local optima, improving the model's resistance to misalignment and addressing the insufficient generalization of baseline models in cross-domain person re-identification.

[0005] A cross-domain person re-identification method based on point-by-point feature matching, characterized by the following steps:

[0006] (1) Introduce IBN-Net into ResNet50 to enable the model to learn the correlation of visual transformations such as style and color, reduce the differences in appearance between individuals, and enhance the cross-domain re-identification performance of the model.

[0007] (2) Then divide the layer 3 feature map of the Resnet50-IBN-Net convolutional network into N sub-feature maps, and convert all sub-feature maps into one-dimensional vectors. Input the converted one-dimensional vectors into the transformer encoder to extract better global and local features.

[0008] (3) Then perform point-by-point matching on the obtained feature map, and output the final similarity result by normalizing and fully connected layer on the final matching vector. Attached Figure Description

[0009] Figure 1 This is a flowchart of the cross-domain pedestrian re-identification method based on point-by-point feature matching according to the present invention.

[0010] Figure 2 The network structure diagram of Resnet50 introducing IBN in this invention;

[0011] Figure 3 This is a diagram of the Transformer encoder network structure of the present invention;

[0012] Figure 4 This is a flowchart of the feature point-by-point matching process of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0014] like Figure 1 As shown, a cross-domain person re-identification method based on point-by-point feature matching is characterized by the following steps:

[0015] (1) Introduce IBN-Net into ResNet50 to enable the model to learn the correlation of visual transformations such as style and color, reduce the differences in appearance between individuals, and enhance the cross-domain re-identification performance of the model.

[0016] (2) Then divide the layer 3 feature map of the Resnet50-IBN-Net convolutional network into N sub-feature maps, and convert all sub-feature maps into one-dimensional vectors. Input the converted one-dimensional vectors into the transformer encoder to extract better global and local features.

[0017] (3) Then perform point-by-point matching on the obtained feature map, and output the final similarity result by normalizing and fully connected layer on the final matching vector.

[0018] The detailed steps are as follows:

[0019] Step 1: Batch Normalization (BN) learns content-related feature information, preserving the differences between individual samples, enhancing the heterogeneity of semantic features, accelerating the model training process, and obtaining discriminative features. Instance Normalization (IN) learns the correlation of visual transformations such as style and color, reducing the differences in appearance between individuals and enhancing the network's ability to learn appearance invariance. It has good robustness to changes in the appearance of the target pedestrian. Therefore, in ResNet50, Instance Normalization (IN) and Batch Normalization (BN) are introduced to form the ResNet50-IBN network to jointly normalize features, solving the problem that using only BN layers is insensitive to feature style transformations. Therefore, the IBN-Net module is introduced into ResNet50, that is, the IN layer is introduced into the first BN layer in the ResNet50 Block, while half of the BN layers are retained, forming the ResNet50-IBN-a network; the IN layer is introduced after the ResNet50 Block residual connection, forming the ResNet50-IBN-b network.

[0020] Step 2: Convert the pedestrian feature map x∈R after CNN convolution. H×W×C Flattened into a two-dimensional sub-feature map N = HW / P 2 N is the feature map divided into two-dimensional sub-feature maps x. p The number of channels is the effective input sequence length of the Transformer, H and W are the height and width of the feature map, respectively, C is the number of channels of the feature map, and (P, P) is the sub-feature map x. p Size, P 2 • C is a one-dimensional vector transformed from all sub-feature maps. The Transformer uses the same dimension D across all layers, thus mapping each sub-feature map to the model's dimension D via a trainable linear projection. The core module of the Transformer is the multi-head attention layer, defined as follows:

[0021]

[0022] Where Q represents the query matrix, K represents the key matrix, and V represents the actual feature matrix. These three parameters are derived from the input feature map z∈R. N×D Generates Q by performing a linear mapping: Q = z × WQ K = z × W K V = z × W V Randomly initialize W Q W K W V Multi-head recognition extracts multiple semantic information through an attention mechanism. The pedestrian feature vector is derived from the above formula, where Q×K... T Calculate the attention matrix and divide by The attention matrix is ​​transformed into a standard normal distribution, and after softmax normalization, the attention matrix is ​​calculated and weighted with V. The dimension of Attention(Q,K,V) is consistent with the dimension of V.

[0023] The multi-head attention mechanism initializes multiple sets of W. Q W K W V This process generates multiple feature vectors (Attention(Q,K,V)), which are then concatenated and their dimensionality reduced by a fully connected layer. This allows for the extraction of more global and local information about the pedestrian. The formula is as follows:

[0024] head i =Attention(QW i Q KW i K yW i V (2)

[0025] MultiHead(Q,K,V)=Concat(head1,...,head H W O (3)

[0026] Step 3: Each image is processed by a backbone CNN network to extract features, and then processed by a transformer to obtain a feature map of size [1, c, h, w], where c is the number of output channels, and h and w are the height and width of the feature map, respectively. This feature map is then cached in a feature cache pool, and matched against all pedestrian feature maps in the pool. At the beginning of training, the cache pool does not contain any pedestrian feature maps; only pedestrian identity features are cached. If a pedestrian's identity feature map already exists, the original pedestrian feature map is replaced with the latest feature map. Once all pedestrians have the latest cached feature maps, the point-by-point matching process for pedestrian feature points begins.

[0027] After performing point-by-point matching of the identity features of two pedestrians, a similarity matrix of size [w×h, w×h] is obtained. The first row of this matrix represents the feature matching result between the first point of the first feature map and all points of the second feature map, and so on for the other rows. The first column represents the feature matching result between the first point of the second feature map and all points of the first feature map, and so on for the other columns. Next, global max pooling is performed in both the horizontal and vertical directions, and the two sets of similarity outputs are concatenated to obtain the optimal local matching vector of size 2×w×h. Finally, batch normalization and a fully connected layer are applied to output the final similarity result.

[0028] Its formal description is as follows, assuming the input network image x∈R b×c×h×w The shape is [b,c,h,w], and each image x i The feature output is y i ∈R c×w×h The cache pool caches the identity features of all pedestrians. Let's assume the feature of the j-th pedestrian is y. j ∈R c×w×h After feature point matching, a one-dimensional vector d of length 2×w×h is obtained. ij Each element in the vector Indicates the input pedestrian feature map y i The k-th feature and the feature map y in the feature cache pool j The best matching similarity of all feature points, and similarly

[0029]

[0030]

[0031] Where D(·) represents the Euclidean distance metric, / / represents floor division, and % represents modulo.

[0032] Put d ij Through full connectivity W f Downsample by 2×w×h, then use the sigmoid activation function to reduce d ij Normalize to the range [0,1] and obtain the final similarity p, p∈R. n Where n is the number of features in the feature buffer pool. Using the vector product in angular form, the input image x... i The probability of being class f is p i (f),

[0033]

[0034] in, While it's a fully connected layer, in the matching process of person re-identification, the angle of the feature vector usually contributes more than the norm of the feature vector. Therefore, the L2 normalization norm is used to constrain it. ||d ij ||=m, During training, the similarity vectors of all matched image pairs are projected onto a hypersphere with approximately the same length, making the model more inclined to use angles for judgment. Input image x i The probability p of being class f i (f) can be rewritten as

[0035]

[0036] Introducing the hyperparameter t increases the loss effect, thereby improving the model's discriminative ability and the ability to aggregate the same category. The scaling factor m can accelerate model convergence.

[0037] This network uses Focal Loss as the loss function. Assuming the input training batch size is [b,c,h,w] and the feature buffer size is [n,c,h,w], the pedestrian similarity p is obtained after feature matching, fully connected layers, and the sigmoid function. i (f) However, during the matching process, there are significantly more negative sample pairs with different identity information than the same positive sample pairs. In order to balance the weights of positive and negative sample pairs, Focal Loss is used to weight the loss.

[0038]

[0039] Where, when p i (f) = 1 indicates a positive sample pair, and when p i When (f) = 0, it represents a negative sample pair. θ1 and θ2 are used to balance positive and negative sample pairs. γ is a focusing parameter used to strengthen the weight of difficult sample data.

[0040] To verify the effectiveness of the method in this invention, Market1501 and DukeMTMC-reID, datasets commonly used in the field of person re-identification, were used for training and testing. During the model testing phase, re-ranking and TLIFT were introduced to further improve accuracy.

[0041] This invention's ablation experiments used ResNet50 as the baseline network. IBN-b indicates the introduction of an IN layer into ResNet50, forming a ResNet50-IBN-b network structure. Similarly, IBN-a and trs indicate the introduction of a Transformer attention module. rerank and TLift represent post-processing methods introduced during the testing phase. Furthermore, comparisons were made with some mainstream algorithms, using Rank-1, Rank-5, Rank-10, and mAP (mean accuracy) as evaluation metrics. Experimental results are shown in Tables 1, 2, and 3. The experimental accuracy is significantly improved compared to the baseline network and other comparative algorithms.

[0042] Table 1 Experimental results of different IBN-Nets

[0043]

[0044] Table 2 Ablation experimental results on Market1501 and DukeMTMC

[0045]

[0046] Table 3 shows the results of the comparison with mainstream algorithms.

[0047]

[0048]

Claims

1. A cross-domain pedestrian re-identification method based on point-by-point feature matching, characterized in that Includes the following steps: (1) Introduce IBN-Net into ResNet50 to enable the model to learn the correlation of style and color visual transformation, reduce the differences in appearance between individuals, and enhance the cross-domain re-identification performance of the model. (2) Then divide the layer 3 feature map of the Resnet50-IBN-Net convolutional network into N sub-feature maps, and convert all sub-feature maps into one-dimensional vectors. Input the converted one-dimensional vectors into the transformer encoder to extract better global and local features. (3) Then, perform point-by-point matching on the obtained feature maps, and output the final similarity result by normalizing and fully connected layers on the final matching vectors; in step (3), each image extracts features through a backbone CNN network, and then passes through a transformer to obtain The feature map of size, where Let h and w be the number of output channels, and h and w be the height and width of the feature map, respectively. The feature map is then cached in a feature cache pool, and matched against all pedestrian feature maps in the pool. At the beginning of training, the cache pool contains no pedestrian feature maps; only pedestrian identity features are cached. If a pedestrian's identity feature map already exists, the original feature map is replaced with the latest one. Once all pedestrians have the latest cached feature maps, the point-by-point matching process for pedestrian feature points begins. After point-by-point matching of the identity features of two pedestrians, the following is obtained: The similarity matrix is ​​then processed, and global max pooling is performed on both the horizontal and vertical directions. The two sets of similarity outputs are concatenated to obtain a 2×w×h optimal local matching vector. Finally, batch normalization and a fully connected layer are applied to output the final similarity result, assuming the input image to the network is... The shape is Each image The feature output is The cache pool caches the identity features of all pedestrians. Let's assume the feature of the j-th pedestrian is... After feature point matching, a length of [length missing] is obtained. a one-dimensional vector Each element in the vector Indicates the input pedestrian feature map The k-th feature and the feature map in the feature cache pool The best matching similarity of all feature points, and similarly ; (1) (2) (3) Where D(·) represents the Euclidean distance metric, Indicates floor removal, Indicates modulo; Bundle Through full connectivity downsampling Then use the sigmoid activation function to multiply it by 100%. Normalize to the [0,1] interval and obtain the final similarity. , Where n is the number of features in the feature buffer pool, the input image is represented by an angular vector product. The probability of being class f is : (4) in, It is a fully connected layer, but in the matching process of person re-identification, the angle of the feature vector usually contributes more than the norm of the feature vector. Normalization paradigm constraint , During training, the similarity vectors of all matched image pairs are projected onto a hypersphere with approximately the same length, making the model more inclined to use angles for judgment. The probability of being class f Can be re-represented as (5) Introducing the hyperparameter t increases the loss effect, thereby improving the model's discriminative ability and the ability to aggregate the same category. The scaling factor m can accelerate model convergence.

2. According to claim 1, in the cross-domain pedestrian re-identification method based on point-by-point feature matching, in step (1), the IBN-Net module is introduced into ResNet50, that is, the IN layer is introduced into the first BN layer in ResNet50 Block, while half of the BN layer is retained to form a ResNet50-IBN-a network; the IN layer is introduced after the ResNet50 Block residual connection to form a ResNet50-IBN-b network.

3. According to claim 1, in the cross-domain person re-identification method based on point-by-point feature matching, in step (2), the feature map of layer 3 of the ResNet50-IBN-Net convolutional network is flattened into N two-dimensional sub-feature maps, and these N two-dimensional sub-feature maps are converted into one-dimensional vectors and input into the transformer encoder. N is the effective sequence length of the input transformer. The transformer encoder mainly contains two sub-layers, namely a multi-head attention layer and a fully connected layer. Each sub-layer is connected by residuals. The core module in the transformer is the multi-head attention layer, and the formula is defined as follows: (6) in, Q represents the query matrix, K represents the key matrix, and V represents the actual feature matrix. Multi-head extraction uses an attention mechanism to extract multiple semantic information. The pedestrian's feature vector is derived from the above formula, where... Calculate the attention matrix and divide by The attention matrix is ​​transformed into a standard normal distribution, and after softmax normalization, the attention matrix is ​​calculated and weighted with V.

Images