Pedestrian re-identification method based on global and salient fine-grained local feature fusion

Abstract

The application considers the problems of pedestrian misalignment and difficulty in capturing local area fine-grained features, and introduces a pedestrian re-identification method of global and salient fine-grained local feature fusion. The features of the image are more efficiently extracted and processed, and the robustness of the model is improved. The attention mechanism is added to make the local features more salient. The problem of difficulty in capturing fine-grained local features is well solved. Then GeMpooling is used to adaptively adjust the degree of pooling, improve the generalization ability and performance of the deep learning model, and introduce the BN layer to speed up the network convergence. Finally, the ID loss and the triple loss are combined to perform feedback iteration, so that the final loss of the model reaches a minimum value. A new effective method for pedestrian re-identification is constructed, which provides a more efficient framework for pedestrian re-identification in practical applications.

Description

[0001] The application relates to the technical field of computer vision, and is a pedestrian re-identification method based on fusion of global features and salient fine-grained local features. BACKGROUND

[0002] Person Re-identification, also known as pedestrian re-identification, is abbreviated as ReID. It is a technology for judging whether a specific pedestrian exists in an image or a video sequence by using computer vision technology. The technology has great application value in criminal investigation and family security management, and is of great research significance. Most existing pedestrian re-identification methods mainly learn discriminative and robust features to cope with various challenges in image recognition, including cluttered backgrounds, different illuminations, posture changes, and different resolutions. Traditional recognition methods mainly extract global information of pedestrians, and perform similarity matching of pedestrians by using global information features. However, due to the complexity of pedestrian data, and the fact that only global information is used for recognition to capture overall context information, it is difficult to capture fine-grained information in easily overlooked local regions, resulting in low accuracy of pedestrian recognition. Therefore, the pedestrian re-identification network needs both global information of pedestrians and salient fine-grained information in local regions to improve the robustness and recognition accuracy of the model. SUMMARY

[0003] In view of the deficiencies of the prior art, the application provides a pedestrian re-identification method based on fusion of global features and salient fine-grained local features. The purpose is to solve the problems of misalignment of pedestrians and difficulty in capturing fine-grained features in local regions, and to extract comprehensive, robust and discriminative features.

[0004] To achieve the above purpose, the application provides a pedestrian re-identification method based on fusion of global features and salient fine-grained local features, which comprises the following steps:

[0005] S1, constructing a network framework, the network being composed of a global branch and a local branch;

[0006] S2, inputting a given pedestrian image into the global branch to extract global features;

[0007] S3, inputting the features into the local branch to extract local features;

[0008] S4, introducing an attention mechanism into the local branch;

[0009] S5, the obtained features being subjected to GeM Pooling and then a BN layer;

[0010] S6, finally calculating ID loss and triplet loss of the obtained features.

[0011] The application provides a pedestrian re-identification method of global and saliency fine-grained local feature fusion.

[0012] The application considers the problems of pedestrian misalignment and difficulty in capturing fine-grained local features, and introduces a pedestrian re-identification method of global and saliency fine-grained local feature fusion. The attention mechanism is added to make the local features more salient. The problem of difficulty in capturing fine-grained local features is well solved. Then, GeM pooling is used to pool the network features, and a BN layer is introduced to speed up the network convergence. Finally, the ID loss and the triplet loss are combined for feedback iteration, so that the final loss of the model reaches a minimum value. Thus, the model is optimized, and the accuracy and robustness of pedestrian re-identification are improved. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0014] Figure 1 It is the total network structure diagram of the pedestrian re-identification method of global and saliency fine-grained local feature fusion in the application.

[0015] Figure 2 It is the structure diagram of the attention mechanism in the application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical scheme and advantages of the application more clear, the following will further describe the application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0017] The application will be described in detail below in combination with the drawings and specific embodiments of the specification. The pedestrian re-identification based on multi-granularity segmentation attention network includes steps S1-S6:

[0018] S1, constructing a network framework, which is composed of a global branch and a local branch;

[0019] S2, inputting a given pedestrian image into the global branch to extract global features;

[0020] S3, inputting the features into the local branch to extract local features;

[0021] S4, attention mechanism is introduced into the local branch;

[0022] S5, the obtained features are subjected to GeM Pooling and then a BN layer;

[0023] S6, finally, the obtained features are used to calculate ID loss and triplet loss.

[0024] The steps are described in detail below.

[0025] In step S1, a network architecture is constructed, which is composed of a global branch and a local branch. The steps are as follows:

[0026] S101, a framework composed of a global branch and a local branch is constructed;

[0027] S102, the extracted features are used for classification.

[0028] In step S2, the given pedestrian image is input into the pedestrian re-identification network to extract features, and the global features of the pedestrian are captured through the global branch. The steps are as follows:

[0029] S201, the backbone network used in the feature extraction part is Resnet50 with four stages. ResNet50 architecture has four residual blocks, which are sequentially passed through, and the channel size of the generated feature map is divided into 256, 512, 1024 and 2048. We set them as the first, second, third and fourth stages in turn. The convolution step of the fourth stage (the last stage) is changed from 2 to 1 to preserve more image details. ResNet50 appears in many classic network architectures and has been widely recognized and proven. Therefore, Resnet50 is used to extract features from pedestrian images.

[0030] S202, given a person image, we first extract a feature map X e R C×H×W . Where C represents the channel dimension, H represents the height, and W represents the width. We use multi-stage feature extraction to enhance feature representation stage by stage. After the last stage, a generalized average pooling is used, and a batch normalization operation is added to speed up the convergence of the network. This branch is mainly used to extract the global features of the pedestrian, and no attention module is added. The main purpose is to make the network more lightweight.

[0031] In step S3, the features obtained through the fourth stage are subjected to the local branch.

[0032] S301, first, the local branch is as shown in the local branch diagram in Figure 1 . This branch mainly obtains the local information of the image.

[0033] S302、We introduce local branch in the fourth stage of backbone network. Generally, local features extracted from images can retain a lot of local information, but they do not pay enough attention to the fine-grained information of the region, resulting in the fine-grained local information with discriminative being ignored, which affects the final feature representation and the performance of human-computer recognition. According to this problem, we introduce a significant fine-grained local branch to better extract local information. We perform a dimension reduction processing on the input feature. First, convolution is performed to make the feature a 512-dimensional feature, then a BN layer is used for batch normalization operation, and finally ReLU is used for activation. After this dimension reduction processing, a 512-dimensional feature vector is obtained. Then the horizontal cutting method is used to extract the more fine-grained local features of pedestrians. We divide the 512-dimensional feature obtained by dimension reduction into four spatial horizontal stripes of the same size, and the dimension of each part is C x H / 4 x W. We denote them as part-level features g1, g2, g3, and g4. The column vectors are respectively subjected to attention mechanism, and the attention of each part is obtained to distinguish similar local information. Then each part-level tensor is averaged into a vector by GeM, and the averaged tensor is connected to obtain a 2048-dimensional feature. Then full connection is performed to generate only one ID prediction loss. Unlike traditional PCB methods, PCB uses multiple ID prediction losses, and each part has an independent ID prediction loss. However, using multiple ID losses may lead to incorrect predictions for pedestrians with certain part-level features.

[0034] In step S4, an attention mechanism is introduced into the local branch.

[0035] S401、Firstly, the framework diagram of the attention mechanism is as Figure 2 shown. The attention mainly focuses on fine-grained local features.

[0036] S402、We introduce an attention module in the local branch, and here we directly use the SE module. The input size of the feature map is C x H x W. First, a global average pooling is performed on it to obtain a feature of size 1 x 1 x C. Then, a full connection layer is used for dimension reduction. Next, ReLU is used for activation. Then, a full connection layer is used, and a sigmid layer is used to obtain the channel weight. The feature with different weights in the channel is obtained, so that important channels get large weights and unimportant channels get small weights. Finally, the learned weight is applied to each channel in the feature map to generate a weighted feature map. In summary, the SE module is a channel attention module, which can be regarded as adaptively learning a weight for each channel of the feature map. Given a feature map X, the channel attention Attention is defined as:

[0037] Attention=Sigmoid(W2ReLU(W1Poolavg (X))) (1)

[0038] where W1 and W2 represent two fully connected operations, Pool avg represent the average pooling layer. The channel attention Attention is applied to the feature map X by channel multiplication. The segmented part-level features g1, g2, g3, g4 are replaced by the original feature map X, and the attention is used to form more significant fine-grained local features.

[0039] In step S5, the passed feature is input into the GeM Pooling and BN layer.

[0040] S501, In order to make the network focus more on the distinctive features, GeM Pooling is adopted in our network, which can enhance feature information, better capture features, and improve the quality of feature expression. In existing pedestrian re-identification research, global average pooling and global maximum pooling are widely used, but they cannot capture discriminative features in specific fields. Global average pooling calculates the average value of the feature map, which will weaken the discriminability of the final representation. While global maximum pooling only calculates the maximum value of each feature map, which may cause the loss of other activation information. We use the more beneficial GeM (Generalized Mean) pooling to replace global average pooling and global maximum pooling. This pooling layer takes X as input and produces a vector f as the output of the pooling process, and its formula is:

[0041]

[0042] where Pk is a pooling hyperparameter that can be learned during backpropagation. When Pk→∞, it approximates the maximum pooling, and when Pk=1, it approximates the average pooling. The final feature vector is composed of a single value from each feature map with generalized mean activation, and its dimension is equal to K, which makes it a compact image representation.

[0043] In the global branch, we use GeM after each stage. The GeM features after each stage are stacked to obtain a more compact vector, which is combined with the classification feature vector generated by the network and input into the ID loss in the training stage. In addition, the feature after the fourth stage is input into the triplet loss after GeM Pooling. In the local branch, we also use GeM after attention to better capture local features.

[0044] S502, The obtained feature is input into the BN layer, i.e. batch normalization, to solve the problem of changing the data distribution of the intermediate layer during training, to prevent gradient vanishing or explosion, prevent overfitting, and speed up convergence.

[0045] In step 6, the obtained feature calculates the ID loss and the triplet loss, which are fed back to the model to iteratively optimize the loss. The specific steps are as follows:

[0046] S601, the loss of the entire network is composed of ID loss and triplet loss. We regard each ID as a different class, and the ID loss can obtain the ID prediction of the image, which is similar to the classification loss. The ID loss is defined as:

[0047]

[0048] where y and p i represent the true ID label and the predicted score of the i-th class respectively. N represents the number of classes, and q i represents the smooth label. ε is the label smoothing parameter, which is set to 0.1.

[0049] In addition, in order to combine the characteristics of pedestrian re-identification and facilitate finding the most similar person to the target image from the image library, a triplet loss is introduced in the training, which is beneficial to increase the distance between classes and reduce the distance within the class. The triplet loss is calculated, and the triplet makes the distance between classes smaller than the distance within the class, and the formula is as follows:

[0050] L tri = [d pos -d neg +m] + (4)

[0051] where d pos and d neg define the distance of positive sample pair and negative sample pair respectively. [·] + represents max(0,x), and m represents the difference boundary of the similarity between the positive sample and the anchor sample, which is set to 0.3. The total loss L is:

[0052] L = L id + L tri (5)

[0053] S602, because the training process of the network model is a process of continuously optimizing the loss, the obtained loss is fed back to the network model for continuous iteration and optimization to reduce the loss, so as to obtain more robust features.

[0054] The application patent improves the precision and robustness of the model by improving the feature extraction method. The application considers the problems of pedestrian misalignment and difficulty in capturing local area fine-grained features, and introduces a pedestrian re-identification method of global and salient fine-grained local feature fusion. The features of the image are more efficiently extracted and processed, and the robustness of the model is improved. The attention mechanism is added to make the local features more salient. The problem of difficulty in capturing fine-grained local features is well solved. Then GeMpooling is used to adaptively adjust the degree of pooling, improve the generalization ability and performance of the deep learning model, and introduce the BN layer to speed up the network convergence. Finally, the ID loss and the triplet loss are combined for feedback iteration, so that the final loss of the model reaches a minimum value. A new effective method for pedestrian re-identification is constructed, which provides a more efficient framework for pedestrian re-identification in practical applications.

[0055] The application also proposes a pedestrian re-identification method of global and salient fine-grained local feature fusion, which comprises:

[0056] A network framework is constructed, and the network is composed of a global branch and a local branch;

[0057] A given pedestrian image is input into the global branch to extract global features;

[0058] The features are input into the local branch to extract local features;

[0059] The attention mechanism is introduced into the local branch;

[0060] The obtained features are subjected to GeM Pooling and then subjected to a BN layer;

[0061] Finally, the obtained features are used to calculate the ID loss and the triplet loss.

[0062] The application also proposes a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to realize the steps of the above-mentioned method.

[0063] The above-mentioned is only the preferred embodiment of the application, and does not limit the patent scope of the application, and any equivalent structural transformation made according to the content of the application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the application.

Claims

1. A pedestrian re-identification method of global and saliency fine-grained local feature fusion, characterized in that, The method is executed by a computer and includes the following steps: S1, a network framework is constructed, which is composed of a global branch and a local branch; S2, a given pedestrian image is input into the global branch to extract global features, and the specific implementation process of S2 is as follows: S201, the backbone network used by the feature extraction part is Resnet50 with four stages, the ResNet50 architecture has four residual blocks, which are sequentially passed through, and the channel size of the generated feature map is divided into 256, 512, 1024 and 2048, which are sequentially set as the first, second, third and fourth stages, and the convolution step of the fourth stage (the last stage) is changed from 2 to 1 to preserve more image details, Resnet50 appears in many classic network architectures and has been widely recognized and proven, so Resnet50 is used to extract features from pedestrian images; S202、Given a person image, we first extract the feature map X ∈ R C×H×W Where C represents the channel dimension, H represents the height, and W represents the width. We use a multi-stage feature extraction method to enhance the feature representation stage by stage. After the last stage, we use generalized average pooling and add a batch normalization operation to speed up the convergence of the network. This branch is mainly used to extract the global features of pedestrians and does not add an attention module to make the network more lightweight. S3, the features are input into the local branch to extract local features, and the specific implementation process of S3 is as follows: S301, this branch mainly obtains local information of the image; S302, we introduce a local branch after the fourth stage of the backbone network, generally, the local features extracted from the image can retain a large amount of local information, but they do not pay enough attention to the fine-grained information of the region, resulting in the loss of discriminative fine-grained local information, which affects the final feature representation and the performance of human-computer recognition, according to this problem, we introduce a salient fine-grained local branch to better extract local information, we perform a dimension reduction processing on the input features, first convolution, so that the features become 512-dimensional, then pass through a BN layer for batch normalization operation, and finally pass through ReLU for activation, after this dimension reduction processing, a 512-dimensional feature vector is obtained, then a horizontal cutting method is used to extract more fine-grained local features of pedestrians, we divide the 512-dimensional features obtained by dimension reduction into four spatial horizontal stripes of the same size, the dimension of each part is CxH / 4xW, we denote them as part-level features g1, g2, g3 and g4, and the column vectors are subjected to attention mechanism, the salient fine-grained local information is obtained to distinguish similar local information, then each part-level tensor is averaged into a vector through GeM, and the averaged tensor is connected to obtain a 2048-dimensional feature, and then full connection is performed to generate only one ID prediction loss, unlike the traditional PCB method, which uses multiple ID prediction losses, each part has an independent ID prediction loss, but using multiple ID losses may lead to incorrect prediction of pedestrians with certain part-level features; S4, attention mechanism is introduced into the local branch; S5, the obtained features are subjected to GeM Pooling and then BN layer; S6, finally, the obtained features are used to calculate ID loss and triplet loss.

2. The global and salient fine-grained local feature fused pedestrian re-identification method of claim 1, wherein, The specific implementation process of S1 is as follows: S101, a framework composed of a global branch and a local branch is constructed; S102, the extracted features are used for classification. 3.The global and salient fine-grained local feature fusion based pedestrian re-identification method of claim 1, wherein, The specific implementation process of S4 is as follows: S401, introduce attention focus on fine-grained local features; S402, we introduce attention module in local branch, here we directly use SE module, the input size is CxHxW feature map, first do a global average pooling, get 1x1xC size feature, then through a fully connected layer, dimension reduction, next, through ReLU activation, then through a fully connected layer, through a sigmoid layer, get channel weight, get channel with different weight features, make important channel get large weight, and unimportant channel get small weight, finally, the learned weight is applied to each channel of the feature map, to generate a weighted feature map, in short, SE module is a channel attention module, which can be regarded as adaptive learning a weight for each channel feature map, given feature map X, channel attention Attention is defined as: Attention = Sigmoid(W2 ReLU(W1 Pool avg (X))) (1) where W1 and W2 represent two full connection operations, Pool avg represent an average pooling layer, and Attention represents channel attention. The channel attention is applied to the feature map X by channel multiplication, and the part-level features g1, g2, g3, and g4 divided by the segmentation are used to replace the original feature map X, and the attention is used respectively to form more significant fine-grained local features. 4.The global and salient fine-grained local feature fusion based pedestrian re-identification method of claim 1, wherein, The specific implementation process of S5 is as follows: S501, in order to make the network focus on the features with distinction, GeM Pooling is used in our network, which can enhance feature information and better capture features to improve the quality of feature expression. In existing pedestrian re-identification research, global average pooling and global maximum pooling are widely used, but they cannot capture discriminative features in specific fields. Global average pooling calculates the average value of feature map, which will weaken the discriminability of the final representation, while global maximum pooling only calculates the maximum value of each feature map, which may lead to the loss of other activation information. We use generalized mean (GeM) pooling instead of global average pooling and global maximum pooling, which is more conducive to network learning. The pooling layer takes X as input and produces a vector f as the output of the pooling process, and its formula is: Where Pk is a pooling hyperparameter that can be learned during backpropagation. When Pk→∞, it is approximately equal to maximum pooling, and when Pk=1, it is approximately equal to average pooling. The final feature vector is composed of single values of each feature map with generalized mean activation, and its dimension is equal to K, which makes it a compact image representation. In the global branch, we use GeM after each stage to stack the GeM features after each stage to get a more compact vector. This vector combines the classification feature vector generated by the network and is input to the ID loss in the training stage. In addition, the features after the fourth stage are input to the triplet loss after generalized average pooling (GeM Pooling). In the local branch, we also use GeM after attention to better capture local features; S502, the obtained features are subjected to BN layer, i.e. batch normalization, to solve the problem of change of intermediate layer data distribution during training, to prevent gradient vanishing or explosion, prevent overfitting and speed up convergence.

5. The global and salient fine-grained local feature fused pedestrian re-identification method of claim 1, wherein, The specific implementation process of S6 is as follows: S601、The loss of the whole network is composed of ID loss and triplet loss. We regard each ID as a different class. The ID loss can get the ID prediction of the image, which is similar to the classification loss. The ID loss is defined as: where y and p i respectively represent the real ID label and the prediction score of the i-th class, N represents the class number, q i represents the smoothed label, and ε is the label smoothing parameter, which is set to 0.

1. In addition, in order to combine the characteristics of pedestrian re-identification, it is convenient to find the most similar person to the target image from the image library, a triplet loss is introduced in the training, which is beneficial to increase the inter-class distance and reduce the intra-class distance. The triplet loss is calculated, and the triplet makes the inter-class distance less than the intra-class distance, and the formula is as follows: L tri = [d pos -d neg +m] + (4) where d pos and d neg define the distance of positive and negative sample pairs, respectively, [·] + denotes max(0, x), m denotes the difference boundary of the similarity between positive samples and anchor samples, and is set to 0.

3. The total loss L is: L = L id + L tri (5) S602、Because the training process of the network model is a process of continuously optimizing the loss, the loss obtained at present is fed back to the network model for continuous iteration and optimization to reduce the loss, so as to obtain more robust features.

Images