Pedestrian re-identification method based on multi-fine-grained feature fusion

Abstract

The invention discloses a pedestrian re-identification method based on multi-fine-grained feature fusion, and the method comprises the steps: firstly obtaining a pedestrian re-identification image data set, and carrying out the preprocessing of original images and labels in the data set; secondly, constructing a pedestrian re-identification network model, and sending the preprocessed picture into the pedestrian re-identification network model to obtain a re-identification result; and finally, designing a loss function, and training the pedestrian re-identification network model by adopting a warmup strategy. And inputting a to-be-recognized image into the trained pedestrian re-recognition network model, and performing a pedestrian re-recognition test. According to the multi-fine-grained feature extraction network, the generalization and robustness of the model are enhanced by capturing rich features from local to global in pedestrian re-identification, and the accuracy of pedestrian identification is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of image retrieval, and in particular relates to a pedestrian re-identification method by fusing multiple fine-grained features. Background Art

[0002] Person re-identification is a research hotspot in the field of computer vision. It aims to solve the problem of matching person identity information between different surveillance cameras and can be applied to urban security monitoring, smart transportation and other fields. With the continuous development of deep learning technology, person re-identification has become one of the issues that people are paying more attention to.

[0003] In the task of person re-identification, one of the main challenges is how to accurately identify the same pedestrian at different times, in different scenes and from different perspectives. In addition, due to the large changes in the appearance of pedestrians, such as clothing, posture, occlusion and other factors, the problem of pedestrian re-identification is more complicated and difficult. Therefore, how to extract feature representation methods with robustness, diversity and effectiveness is an important research topic. In recent years, deep learning has made significant progress in the task of pedestrian re-identification, such as using convolutional neural networks (CNN) for end-to-end feature learning and robustness modeling, and using methods such as transfer learning, multimodal learning and weakly supervised learning to further improve the accuracy and efficiency of pedestrian re-identification. At the same time, more and more studies have proposed a variety of innovative models and algorithms for pedestrian re-identification. Due to different camera parameters and shooting angles, pedestrian images have large differences in background, resolution, perspective and posture. How to extract discriminative features is the key to solving this problem.

[0004] The key to multi-granularity feature extraction is to capture both macroscopic and microscopic features of pedestrians and effectively integrate these features together to improve the accuracy and robustness of pedestrian re-identification. After multi-granularity feature extraction, an effective fusion strategy is needed to integrate features of different granularities. Common fusion methods include feature-level fusion and decision-level fusion, which can combine features of different granularities at different stages. As research deepens, this field will continue to develop new technologies and methods to meet the complex challenges in practical applications. Summary of the invention

[0005] Based on the above analysis, the present invention proposes a pedestrian re-identification method with multi-fine-grained feature fusion, which includes feature erasing branches, global branches and local branches to provide a network structure with high accuracy and generalization. To solve the following problems in the current pedestrian re-identification method: In the camera scene, pedestrians are blocked by other objects, resulting in the loss of some feature information, affecting the recognition accuracy; pedestrian posture changes are likely to interfere with recognition; factors such as changes in light intensity and low camera resolution will reduce the resolution of the image, resulting in reduced pedestrian distinguishability.

[0006] In order to solve the above problems, the present invention provides a pedestrian re-identification method based on multi-fine-grained feature fusion. The overall scheme includes two networks: data enhancement and pedestrian recognition. The specific steps are as follows:

[0007] Step 1: Obtain a person re-identification image dataset and preprocess the original images and labels in the dataset. The preprocessing process includes cropping, random rotation, and random erasing enhancement. Random erasing enhancement introduces random, local occlusions to train the model to increase the robustness to occlusion and noise. This helps improve the adaptability of the model in real scenes and makes it more capable of handling occlusion, lighting changes, and other noise.

[0008] Step 2: Construct a person re-identification network model, send the preprocessed image into the person re-identification network model, and obtain the re-identification result.

[0009] The pedestrian re-identification network model is constructed by a feature pyramid network structure, a detail feature extraction module, a semantic feature extraction module and a feature fusion module. The specific process is as follows:

[0010] Step 2.1, send the preprocessed image into the feature pyramid network structure, and obtain the feature map; the feature pyramid network structure includes five layers of convolution: layer0, layer1, layer2, layer3, layer4.

[0011] Step 2.2, send the feature map F3 output by layer 3 of the feature pyramid structure to the detail feature extraction module. The detail feature extraction module contains two branches. The first branch divides the feature F3 horizontally into five blocks and adds average pooling to obtain feature F3'. The other branch performs average pooling and maximum pooling on feature F3 respectively, and adds and fuses them item by item to obtain feature F3", calculates the difficult triplet loss based on F3", and sends F3" to the classifier to obtain ID loss.

[0012] Step 2.3, send the feature map F4 output by the feature pyramid layer 4 to the semantic feature extraction module; the semantic feature extraction module is divided into two branches; the first branch performs average pooling and maximum pooling respectively, adds and fuses the results of average pooling and maximum pooling item by item to obtain the feature F4' computationally difficult triplet loss, and sends the feature F4' to the classifier to obtain the ID loss; the second branch divides the feature map F4 horizontally into 4 blocks, and performs average pooling to obtain the feature F4".

[0013] Step 2.4: Send the feature maps F4” and F3’ output by the second branch of the detail feature module and the second branch of the semantic feature extraction module to the feature fusion module. After element-by-element addition, they are passed through batch normalization and the classifier of the fully connected layer to calculate the ID loss.

[0014] Step 2.5: Average and merge the similarity score lists obtained by the above three classifiers to generate the final similarity score list and output the recognition result.

[0015] Step 3: Design a loss function and use the warmup strategy to train the person re-identification network model; input the image to be identified into the trained person re-identification network model to perform a person re-identification test.

[0016] The beneficial effect of the present invention is that the multi-fine-grained feature extraction network captures rich features from local to global in pedestrian re-identification, enhances the generalization and robustness of the model, improves the retrieval efficiency, and significantly improves the accuracy of pedestrian recognition and the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The overall network structure diagram of the pedestrian re-identification network of the present invention;

[0018] Figure 2 It is a schematic diagram of characteristic erasure of the present invention;

[0019] Figure 3 It is a schematic diagram of the change of learning rate of the present invention;

[0020] Figure 4 This is the visualization result of the present invention on Market1501. DETAILED DESCRIPTION

[0021] The structure diagram of a pedestrian re-identification method based on multi-fine-grained feature fusion is as follows: Figure 1 As shown in the figure, high-level features have rich semantic information, and low-level features have rich detail information. Therefore, the multi-fine-grained feature network is designed with two modules to extract multi-level features of semantic information and detail information of pedestrian images respectively. Specifically, the feature map extracted by the semantic feature extraction module is denoted as F, and the feature map extracted by the detail feature extraction branch is denoted as F′.

[0022] Obtain a person re-identification image dataset and preprocess the original images and labels in the dataset. The preprocessing process includes cropping, random rotation, and random erasing enhancement. Random erasing enhancement introduces random, local occlusions to train the model to increase the robustness to occlusion and noise. This helps improve the adaptability of the model in real scenes and makes it more capable of handling occlusion, lighting changes, and other noise.

[0023] In order to extract multiple fine-grained features, the present invention introduces a global branch to extract the overall contour features of the pedestrian image, and introduces a local branch to extract detail features (such as watches, clothing patterns, shoe colors, etc.). Global maximum pooling is usually used to extract global features. In order to better obtain the contour of the pedestrian image, the global average pooling and global maximum pooling of different scales are used in series to obtain the global features of the semantic feature extraction branch and the detail feature extraction branch. and

[0024]

[0025] Where f(,) is a concatenation operation, maxp is the global maximum pooling, and avgp is the global average pooling.

[0026] The local branch of the semantic feature extraction module divides the feature map into four blocks, namely Each block is pooled by 2×2 global average to obtain features, and finally they are concatenated to obtain the final features. The local branch of the detail feature extraction module divides the feature map into five blocks, namely Each block is pooled by 2×2 global average pooling to obtain features, and finally they are concatenated to obtain the final features. The detail feature extraction branch is located at the low level, and the feature map contains richer detail information. Therefore, dividing the feature map more finely is conducive to obtaining richer image details.

[0027]

[0028] Feature Fusion Model:

[0029] An efficient feature fusion method is proposed. First, two feature maps are fused and Perform element-by-element addition to achieve initial feature fusion. The goal of this step is to combine and low level The feature information of the two networks is combined to obtain a more comprehensive representation. Then, Batch Normalization (BN) and ReLU operations are applied to process the fused features. Batch Normalization helps to normalize the feature distribution, accelerate the convergence process, and improve the generalization ability of the model. The ReLU operation introduces nonlinear mapping to enhance the expression ability of the network.

[0030] In order to preserve the information of the original features, a skip connection is introduced. This connection allows the original features to be element-by-element added to the features processed by BN and ReLU. This skip connection design helps alleviate the gradient vanishing problem and enables the network to better learn the details of the original features.

[0031] In summary, the feature fusion method achieves lightweight but effective feature fusion through step-by-step operations, namely element-by-element addition, BN and ReLU processing. While retaining the original feature information, it improves the model's adaptability to complex scenes and achieves significant performance improvements in tasks such as pedestrian re-identification.

[0032] Random Erase Enhancement:

[0033] Figure 2 The data augmentation method adopted in this study is shown. Random erasing augmentation is an effective data augmentation technique. Its main purpose is to introduce random, local occlusions to train the model to increase the robustness to occlusion and noise. This helps to improve the adaptability of the model in real scenes and make it more capable of handling occlusion, lighting changes and other noise.

[0034] In addition, random erasing can also mitigate the risk of overfitting by forcing the model to focus on different parts and prevent over-reliance on certain local features. It also increases the diversity of the data, expands the training dataset, and improves the generalization ability of the model. By forcing the model to recognize under occlusion, random erasing also promotes feature learning and helps to generate more discriminative pedestrian features, thereby improving model performance. In summary, random erasing enhancement plays a key role in pedestrian re-identification, providing more challenging and diverse data for model training, thereby improving the robustness and performance of the model.

[0035] Involving the loss function Loss, the warmup strategy is used to train the pedestrian re-identification network model, and the learning rate changes as follows Figure 3 As shown in the figure, the image to be identified is input into the trained person re-identification network model to perform the person re-identification test.

[0036] Loss function:

[0037] The present invention adopts the cross entropy loss commonly used in computer vision classification tasks as the recognition loss. By using the softmax function, the probability is calculated in the following way.

[0038]

[0039] is the predicted probability of the softmax function, and n is the number of samples. The softmax cross entropy loss L is used. id As the recognition loss, the formula is as follows.

[0040]

[0041] in represents the output predicted by the softmax function, p(x i ) represents the target probability, and n is the number of samples.

[0042] The difficult triplet loss is an improved version of the triplet loss. The traditional triplet randomly samples three pictures from the training data. Although this approach is relatively simple, most of the sampled samples are simple and easy to distinguish. If a large number of training sample pairs are simple sample pairs, it will not be conducive to the network learning better representations. Using more difficult samples to train the network can improve the generalization ability of the network. The difficult triplet loss is for each training batch, randomly selects P pedestrians with IDs, and randomly selects K different pictures for each pedestrian, that is, a batch contains P×K pictures. After that, for each picture a in the batch, you can select the most difficult positive sample and the most difficult negative sample to form a triplet with a:

[0043]

[0044] Where α is a manually set threshold parameter. The difficult triplet loss calculates the Euclidean distance between a and each image in the batch in the feature space, and then selects the positive sample p with the farthest distance (least similar) to a and the negative sample n with the closest distance (most similar) to calculate the triplet loss.

[0045] In the ReID task, the recognition loss and the difficult triplet loss are combined to give full play to their respective advantages. In this study, the following loss function combination is used:

[0046] L total =λL id +(1-λ)L th #(8)

[0047] Where λ is the weight parameter of the two functions. In the experiment, when λ = 0.6, the network model performs best.

[0048] The metric learning method is implemented by embedding two feature vectors with the same ID. Generally, in pedestrian re-identification (ReID) models, the methods used to measure distance include Euclidean distance, Hamming distance and Manhattan distance. The cosine distance metric is mainly used in character image retrieval tasks. The cosine distance metric parameterizes the classification system output by softmax to learn the optimized feature space. In the present invention, for the feature vectors A and B calculated from two input images, their distance is calculated as follows:

[0049]

[0050] The Cumulative Matching Characteristic Curve (CMC Curve) is an indicator used to evaluate the performance of pedestrian re-identification algorithms. The curve shows the matching accuracy of the algorithm by gradually increasing the ranking in the matching results. The CMC curve reflects the performance of the algorithm under different rankings and intuitively presents the accuracy of the model in the first k matches. On the curve, Rank-1 accuracy is one of the key points, indicating the accuracy under the highest ranking in the matching results:

[0051]

[0052] The CMC curve is obtained by It is calculated by adding and dividing by the total number of query images, usually expressed as Rank-k. For example, Rank-1 accuracy represents the probability of correctly matching the first target in the matching list.

[0053] Mean Average Precision (mAP) is one of the commonly used evaluation metrics in person re-identification tasks. It comprehensively evaluates the performance of the algorithm by calculating the average precision of different identities. For each identity, the matching results of the algorithm are sorted and then the accuracy is calculated. The average precision takes into account the balance between different identities in the matching results, assigning equal weight to each identity. Therefore, mAP provides a comprehensive evaluation of the performance of the algorithm on the entire test set, which is particularly useful for datasets with diverse identity distributions. In mAP, a higher value indicates that the algorithm performs better in identifying different identities:

[0054]

[0055] Where Q represents the total number of queries in the test set, q represents the index of the query, and AveP represents the precision of query q.

[0056] In this paper, in order to evaluate the actual performance of the proposed MLF-NET model and compare it with other advanced methods, Rank-1 and mAP indicators are used to evaluate the accuracy.

[0057] In order to verify the effectiveness of the multi-fine-grained feature network, ablation experiments are conducted to evaluate the effectiveness of random erasure enhancement and multi-fine-grained feature modules. As shown in Table 1, when only the random erasure enhancement strategy is adopted on the Baseline, the Rank-1 and mAP on the Market1501 dataset are improved by 0.5% and 0.9%, respectively, and on the DukeMTMC-reID dataset by 0.9% and 0.5%; when only the MLF module is added, the Rank-1 and mAP on the Market1501 dataset are improved by 1.0% and 2.7%, respectively, and on the DukeMTMC-reID dataset by 2.7% and 1.8%. When both random erasure enhancement and MLF-NET modules are added to the Baseline, the model achieves the best performance, which is significantly improved over the Baseline. Figure 4 The visualization results of the proposed method on the Market1501 dataset are shown. The model has good pedestrian matching results in environments with occlusion, complex background, and pedestrian posture changes. Random erasing enhancement improves the generalization of the network by artificially increasing occlusion, and prevents over-reliance on certain local features by forcing the model to focus on different parts; the MLF module obtains the discriminative feature representation of pedestrian images by extracting multi-granularity information. In addition, FFB effectively fuses the fine-grained information of multi-level features to obtain robust features. These results show that the proposed method can effectively improve the performance of the Re-ID model and achieve excellent recognition results.

[0058] Table 1 Comparison of recognition performance of different networks

[0059]

Claims

1. A pedestrian re-identification method based on multi-fine-grained feature fusion, characterized in that: The following steps are involved: Step 1: Obtain a person re-identification image dataset and pre-process the original images and labels in the dataset; Step 2: construct a person re-identification network model, send the pre-processed image to the person re-identification network model, and obtain the re-identification result; Step 3: Design a loss function and use the warmup strategy to train the person re-identification network model; input the image to be identified into the trained person re-identification network model to perform a person re-identification test.

2. According to the method of claim 1, wherein: Step 1: The preprocessing process includes cropping, random rotation and random erasing enhancement.

3. The pedestrian re-identification method based on multi-fine-grained feature fusion according to claim 2 is characterized in that: The pedestrian re-identification network model is constructed by a feature pyramid network structure, a detail feature extraction module, a semantic feature extraction module and a feature fusion module.

4. The pedestrian re-identification method based on multi-fine-grained feature fusion according to claim 3 is characterized in that: The specific implementation process of the pedestrian re-identification network model is as follows: Step 2.1: Send the preprocessed image into the feature pyramid network structure and obtain the feature map; The feature pyramid network structure includes five layers of convolution: layer0, layer1, layer2, layer3, layer4; Step 2.2, the feature map F3 output by the layer 3 of the feature pyramid structure is sent to the detail feature extraction module. The detail feature extraction module contains two branches. The first branch divides the feature F3 horizontally into five blocks and adds average pooling to obtain the feature F3'. The other branch performs average pooling and maximum pooling on the feature F3 respectively, and adds and fuses them item by item to obtain the feature F3", calculates the difficult triplet loss from F3", and sends F3" to the classifier to obtain the ID loss; Step 2.3, the feature map F4 output by the feature pyramid layer 4 is sent to the semantic feature extraction module; The semantic feature extraction module is divided into two branches. The first branch performs average pooling and maximum pooling respectively, adds and fuses the results of average pooling and maximum pooling item by item, obtains the feature F4' computationally difficult triplet loss, and sends the feature F4' to the classifier to obtain the ID loss. The second branch divides the feature map F4 horizontally into 4 blocks and performs average pooling to obtain the feature F4". Step 2.4, the feature maps F4" and F3' output by the second branch of the detail feature module and the second branch of the semantic feature extraction module are sent to the feature fusion module, and after element-by-element addition, they are passed through batch normalization and the classifier of the fully connected layer to calculate the ID loss; Step 2.5: Average and merge the similarity score lists obtained by the above three classifiers to generate the final similarity score list and output the recognition result.

5. The pedestrian re-identification method based on multi-fine-grained feature fusion according to claim 4 is characterized in that: The difficult triplet loss is specifically implemented as follows: The difficult triplet loss is an improvement of the triplet loss. For each training batch, P pedestrians with IDs are randomly selected, and K different pictures are randomly selected for each pedestrian, that is, a batch contains P×K pictures. Then, for each picture a in the batch, the most difficult positive sample and the most difficult negative sample are selected to form a triplet with a. The formula for the difficult triplet loss is as follows: Where α is the set threshold parameter, and the difficult triplet loss calculates the Euclidean distance between a and each image in the batch in the feature space, and then selects the positive sample p farthest from a and the negative sample n closest to a to calculate the triplet loss.

Images