Pedestrian re-identification method based on shielding enhancement and mask learning

Abstract

The invention discloses a pedestrian re-identification method based on occlusion enhancement and mask learning, and the method specifically comprises the following steps: 1, obtaining a data set, dividing the data set into a training set, a gallery set and a query set, and enabling the gallery set and the query set to form a verification set; step 2, performing occlusion enhancement on the images in the training set, and generating mask images; step 3, carrying out image scaling and normalization processing on the original image in the training set and the image generated after shielding enhancement at the same time to obtain a preprocessed image; step 4, constructing a pedestrian re-identification model based on shielding enhancement and mask learning; 5, training the model constructed in the step 4 by adopting the image preprocessed in the step 3; and step 6, performing zooming and normalization processing on the images in the verification set, inputting the images into the model trained in the step 5, and finally obtaining a retrieved pedestrian list. The method provided by the invention can improve the robustness of pedestrian re-identification in the sheltered scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and relates to a pedestrian re-identification method based on occlusion enhancement and mask learning. Background Art

[0002] Person Re-identification (ReID) is a computer vision task that aims to identify images of the same person across camera views. It has a wide range of applications, including video surveillance, smart security, and pedestrian tracking. However, in practical applications, pedestrian images may be subject to various interferences, the most challenging of which is the occlusion problem. When pedestrians are partially occluded by other objects (such as vehicles, trees, or other pedestrians), traditional person re-identification algorithms often have difficulty accurately identifying the target person. In this case, the person re-identification task is called occluded person re-identification.

[0003] Traditional person re-identification methods usually rely on hand-crafted features and metric learning methods. For example, color histograms, texture features, SIFT (Scale Invariant Feature Transform), etc. are all commonly used hand-crafted features. These methods perform well in simple scenes, but they do not work well when dealing with complex environments (such as occlusion, lighting changes, etc.).

[0004] Compared with traditional methods, deep learning methods, especially convolutional neural networks (CNNs), have shown powerful performance in pedestrian re-identification.

[0005] In person re-identification, the use of external models such as human body parsing and key point detection can assist in extracting human body structure information and improve recognition accuracy. However, these methods also have some disadvantages. First, the introduction of external models increases the complexity and computational overhead of the system, making the real-time performance of the model poor. Second, external models often have domain deviations from person re-identification models. If the reasoning results of the external models are wrong, the person re-identification results may be unreliable. In addition, these external models usually require pre-training and have poor adaptability to data distribution and scene changes, which limits their promotion in practical applications. Summary of the invention

[0006] The purpose of the present invention is to provide a pedestrian re-identification method based on occlusion enhancement and mask learning. The method can generate an accurate occlusion mask as a supervision signal of the model by performing pixel-by-pixel subtraction between the generated occlusion-enhanced image and the original image, thereby improving the robustness of pedestrian re-identification in occluded scenes.

[0007] The technical solution adopted by the present invention is a pedestrian re-identification method based on occlusion enhancement and mask learning, which specifically includes the following steps:

[0008] Step 1, obtain a data set, and divide the data set into a training set, a gallery set, and a query set, where the gallery set and the query set constitute a validation set;

[0009] Step 2, perform occlusion enhancement on the images in the training set and generate a mask image;

[0010] Step 3, the original image in the training set and the image generated after occlusion enhancement are simultaneously subjected to image scaling and normalization processing to obtain a preprocessed image;

[0011] Step 4: Build a pedestrian re-identification model based on occlusion enhancement and mask learning;

[0012] Step 5, using the image preprocessed in step 3 to train the model constructed in step 4;

[0013] Step 6: After scaling and normalizing the images in the validation set, input them into the model trained in step 5 to finally obtain the retrieved pedestrian list.

[0014] The present invention is also characterized in that:

[0015] The specific process of step 2 is:

[0016] Step 2.1, randomly sample the images in the Occluded-DukeMTMC query set to obtain image X, segment image X, manually select the occlusion blocks B formed by the occlusion area, and form an occlusion block image set S. The segmented occlusion block images are divided into two categories according to common occlusion and pedestrian occlusion, and the prior position information is added to the image file name respectively;

[0017] Step 2.2: randomly generate a random number T between 0 and 1, and perform the training on the original image X ori Perform occlusion enhancement. If T < 0.5, perform ordinary occlusion enhancement; if T ≥ 0.5, perform pedestrian occlusion enhancement to obtain the occlusion enhanced image X occ ;

[0018] Step 2.3, subtract the original image and the occlusion-enhanced image pixel by pixel to obtain the mask image X of the occluded area mask , X mask The generation process of is shown in the following formula (1):

[0019] X mask =X occ -X ori (1)

[0020] Among them, X mask Represents the generated mask image.

[0021] The specific process of step 3 is:

[0022] Step 3.1: Use bicubic interpolation to interpolate image X ori and X occ Zoom in and get the zoomed image X ori_r and X occ_r ;

[0023] Step 3.2, for image X ori_r and X occ_r Normalize them separately to get the normalized image X ori ' and X occ '.

[0024] In step 4, the pedestrian re-identification model based on occlusion enhancement and mask learning includes the Vision Transformer backbone network, mask predictor, Transformer encoder, feature context enhancement module and classifier;

[0025] Image X ori ' and X occ 'After feature extraction through the Vision Transformer backbone network, global features and local features are obtained. Then, the local features are predicted by the mask predictor to obtain four regional masks. The predicted four regional masks are applied to the local feature sequence to obtain four local features that remove occlusion interference. The maximum value of the four regional masks along the channel is obtained to obtain the foreground mask M F , the foreground mask is applied to the local feature sequence to obtain the global feature without occlusion interference, and then the global and local features without occlusion interference are input into the Transfomer encoder for feature refinement to obtain more representative features, and then the global and local features are input into the feature context enhancement module for bidirectional enhancement to obtain a more robust feature representation, and then the enhanced global and local features are input into the classifier for classification.

[0026] The specific process of step 4 is:

[0027] Step 4.1, use VisionTransformer as the backbone network for feature extraction to obtain the extracted feature vector E' ori and E'' occ ;

[0028] Step 4.2, extract the characteristic sequence E'' ori and E'' occ The class tag CLS ori and CLS occ As a global feature, the feature sequence E'' ori and E'' occThe feature vector other than the class label represents the local feature vector sequence, where E'' ori The local eigenvector sequence of p_ori , E'' occ The local eigenvector sequence of p_occ , f p_ori and f p_occ The mask prediction is performed through the mask predictor. The mask prediction process is shown in formula (2):

[0029]

[0030] Among them, Reshape means changing the shape of the feature to the specified shape, Conv means 1*1 convolution, BN means batch normalization layer, ReLU means ReLU activation function, SoftMax means SoftMax function, represents f p_ori Four region masks predicted by the mask predictor, represents f p_occ Four region masks predicted by the mask predictor;

[0031] Step 4.3, for the four region masks predicted in step 4.2 Taking the maximum value along the channel gives Also take the maximum value along the channel to get Then the class label CLS taken out in step 4.2 is ori and CLS occ As a global feature, Acts as foreground mask with CLS ori ,Will Acts as foreground mask with CLS occ , we get the global feature Z of occlusion suppression g_ori and Z g_occ , the above process is expressed by formula (3):

[0032]

[0033] Among them, TransformerBlock represents the Transformer encoder, cat represents the feature concatenation operation, max represents the maximum value operation, and [0] represents the extraction of class labels as global feature representation. and Represent the foreground masks of the original image and the enhanced image, respectively, Z g_ori and Z g_occ Represents the global features that remove occlusion interference;

[0034] Step 4.4, mask the four regions Acting on the local eigenvector sequence fp_ori , and obtain four local features to remove occlusion interference Will The four region masks act on the local feature vector sequence f p_occ , and obtain four local features to remove occlusion interference Then the local features and global features CLS ori Splicing, local features and global features CLS occ Splicing, and then sending it to the Transformer encoder for feature refinement, and finally obtaining the refined local features and The above process is expressed by formula (4):

[0035]

[0036] in, and Represent the i-th local features of the original image and the enhanced image respectively, and Respectively and The local features after occlusion suppression and feature refinement, cat represents the feature concatenation operation, TransformerBlock represents the Transformer encoder, and [0] represents the class label extraction operation;

[0037] Step 4.5: The global feature Z obtained in step 4.3 is g_ori And the local features obtained in step 4.4 And the global feature Z obtained by enhancing the image g_occ and local features The feature context enhancement modules are input separately for bidirectional enhancement.

[0038] Step 4.6, classify the enhanced local features and global features obtained in step 4.5 through classifiers respectively to obtain the final classification results.

[0039] The specific process of step 4.4 is:

[0040] Step 4.4.1, first use the global feature Z g_ori For local features Enhancement is completed in the non-local attention module to obtain enhanced local features The above process is expressed by the following formula (8):

[0041]

[0042] Among them, δ i represents a two-dimensional convolution of size 1*1, f p_ori_1 i yes Through convolution δ 1 The result after mapping, f p_ori_2 ' and f p_ori_2 "It's Z g_ori Through convolution δ 2 and δ 3 The result after mapping is It means f p_ori_1 i and f p_ori_2 "Similarity vector, is the enhanced local feature, and the symbol ⊙ represents transposed multiplication;

[0043] First use the global feature Z g_occ For local features Enhancement is completed in the non-local attention module to obtain enhanced local features The above process is expressed by the following formula (9):

[0044]

[0045] Among them, f p_occ_1 i yes Through convolution δ 1 The result after mapping, f p_pcc_2 ' and f p_occ_2 "It's Z g_ori Through convolution δ 2 and δ 3 The result after mapping is It means f p_occ_1 i and f p_occ_2 "Similarity vector, is the enhanced local feature;

[0046] Step 4.4.2: The enhanced local features obtained in step 4.4.1 are For the global feature Z g_ori Enhanced, the enhanced global feature Z is obtained g_ori ', this process is expressed by the following formula (10):

[0047]

[0048] Among them, f g_ori_1 It's Z g_ori Through convolution δ 1 The result after mapping is and yes Through convolution δ 2 and δ 3 The result after mapping is It means f g_ori_1 and The vector of similarity, Z g_ori ' is the enhanced local feature;

[0049] Similarly, the enhanced local features obtained in step 4.4.1 are For the global feature Z g_occ Enhanced, the enhanced global feature Z is obtained g_occ ', the above process is expressed by the following formula (11):

[0050]

[0051] Among them, f g_occ_1 It's Z g_occ Through convolution δ 1 The result after mapping is and yes Through convolution δ 2 and δ 3 The result after mapping is It means f g_occ_1 and The vector of similarity, Z g_occ ' is the enhanced local feature, and the symbol ⊙ represents transposed multiplication.

[0052] During the training process in step 5, the loss function used is shown in the following formula (12):

[0053] Loss = L id +L tri +λL mask (12)

[0054] Among them, L id is the ID loss function, L tri is the triplet loss, L mask represents the Mask_DIV loss function, and λ represents the weight of the Mask_DIV loss function in the total Loss;

[0055] Among them, ID loss L id The following formula (13) is used for calculation:

[0056]

[0057] Among them, n is the number of samples in each batch training, N is the total number of batches, p(y i |x i) is the input image x i and the class label y i , after SoftMax classification, x i Identified as y i Probability of class;

[0058] Triplet loss L tri The following formula (14) is used for calculation:

[0059]

[0060] Among them, d(a i ,p i ) represents the anchor point a i With positive sample p i The distance between i ,n i ) represents the anchor point a i With negative samples n i The distance between them, margin is a hyperparameter, and M is the total number of triplets;

[0061] Mask_DIV loss function L mask The following formula (15) is used for calculation:

[0062]

[0063] Where K is the number of masks, M i and M j are the outputs of the i-th and j-th masks respectively, and KL represents the KL divergence.

[0064] The specific process of step 6 is:

[0065] Step 6.1: Use bicubic interpolation to transform image X val Scaling is performed to obtain the scaled image X val_r ;

[0066] Step 6.2, for image X val_rr Normalize to get the preprocessed image X val ';

[0067] Step 6.3: image X processed in step 6.2 val ', put it into the model trained in step 5, and finally get the retrieved pedestrian list.

[0068] The beneficial effect of the present invention is that the method proposed in the present invention has the ability to perceive pedestrian occlusion, and can efficiently remove the adverse effects of occlusion areas on model feature extraction and accuracy. The image enhancement method of occlusion simulation proposed in the present invention can increase the occlusion data during training and generate masks to supervise the mask prediction results of the model, thereby more accurately perceiving the occluded area. By adding the Mask_DIV loss function, the model can learn more accurate and non-redundant mask areas. And the added feature context enhancement module can bidirectionally enhance the representation capabilities of global features and local features, and improve the accuracy of the model. The verification on the public data set shows that the method of the present invention has higher indicators and excellent performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a flow chart of a pedestrian re-identification method based on occlusion enhancement and mask learning of the present invention;

[0070] Figure 2 A schematic diagram of occlusion simulation in the pedestrian re-identification method based on occlusion enhancement and mask learning of the present invention;

[0071] Figure 3 A schematic diagram of the structure of a pedestrian re-identification model based on occlusion enhancement and mask learning constructed in the pedestrian re-identification method based on occlusion enhancement and mask learning of the present invention;

[0072] Figure 4 Schematic diagram of the backbone network structure of Vision Transformer in the pedestrian re-identification method based on occlusion enhancement and mask learning of the present invention;

[0073] Figure 5 It is a structural diagram of a mask predictor in a pedestrian re-identification method based on occlusion enhancement and mask learning of the present invention;

[0074] Figure 6 It is a structural schematic diagram of a feature context enhancement module in a pedestrian re-identification method based on occlusion enhancement and mask learning of the present invention;

[0075] Figure 7 This is a visualization effect diagram comparing the pedestrian re-identification method based on occlusion enhancement and mask learning of the present invention with other methods. DETAILED DESCRIPTION

[0076] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0077] Example 1

[0078] The pedestrian re-identification method based on occlusion enhancement and mask learning of the present invention is as follows: Figure 1 As shown, the specific steps include:

[0079] Step 1: The input data is the Occluded-DukeMTMC dataset, which is formed by selecting the occluded data in the DukeMTMC-reID dataset. It includes three parts: training set, gallery set, and query set. The training set contains 15,618 images of 702 different pedestrians, the query set contains 2,210 images of 519 people, and the gallery set contains 17,661 images of 1,110 different pedestrians. In the training phase, the training set is used for training. In the validation phase, the query set and gallery set form the validation set, and the query set is used as the target pedestrian to be queried to query in the gallery set.

[0080] Step 2: Perform occlusion enhancement on the image according to the occlusion enhancement strategy proposed by the present invention. The specific process is as follows: Figure 2 As shown. First: randomly sample the images in the query set of Occluded-DukeMTMC to obtain the original image X, segment X, manually select the occlusion block B formed by the occlusion area, form an occlusion block image set S, add the prior position information to the occlusion block image file name, and then generate a random number T between 0 and 1. If T<0.5, perform ordinary occlusion enhancement, and if T≥0.5, perform pedestrian occlusion enhancement. Then, the original image X in the training set is ori After enhancement, the image X is generated occ , X occ and X ori Subtract pixel by pixel to get X mask , the detailed steps are described as follows:

[0081] Step 2.1, randomly sample the images in the query set of Occluded-DukeMTMC to obtain image X, segment X, and manually select the occlusion block B formed by the occlusion area to form the occlusion block image set S. The segmented occlusion block images are divided into two categories according to common occlusion and pedestrian occlusion, and the prior position information is added to the image file name. For common occlusion, umbrellas are usually blocked above the image (upper left corner or upper right corner), usually aligned with the entire image above, so the suffix "_1" is added to the occlusion block image file name of the umbrella. The occlusion images of road signs and vehicles are usually aligned with the entire image below (lower left corner or lower right corner), so the suffix "_2" is added to the occlusion block image file name of road signs and vehicles.

[0082] Step 2.2: randomly generate a random number T between 0 and 1, and perform the training on the original image X ori Perform occlusion enhancement. If T < 0.5, perform ordinary occlusion enhancement as shown in step 2.2.1. If T ≥ 0.5, perform pedestrian occlusion enhancement as shown in step 2.2.2.

[0083] Step 2.2.1: Ordinary occlusion enhancement: Randomly select an ordinary occlusion block image OP from the occlusion block image set S obtained in step 2.1, read the occlusion block image OP and then read its name suffix. If the suffix is ​​"_1", align the upper right corner of the occlusion block image OP with the original image X. ori Align the upper right corner and then paste to generate the enhanced image X occ .

[0084] Step 2.2.2, pedestrian occlusion enhancement: Randomly select a pedestrian occlusion block image SP from the occlusion block image set S, first according to the width H of the occlusion block SP i and the original image X ori Width H O Calculate the ratio R of the two w , if R w If the ratio is less than R, no processing is performed. w If the ratio is greater than or equal to R, the occlusion block OP is scaled to R times the original size, that is, the size of the occlusion block OP is scaled to 80% of the original size. In the present invention, R is set to 0.8. The effect of the value of R on the performance is shown in Table 1. As can be seen from Table 1, the performance is best when R is 0.8. The purpose of this is that the occlusion area in the real pedestrian occlusion situation is generally not too large, and the target pedestrian will have a part of the visible area. In order to prevent the occlusion block from blocking the entire target pedestrian, it is scaled. Then the lower left corner of the occlusion block image is aligned with the original image X ori Align the lower left corner and paste to form the occlusion enhancement image X occ .

[0085] Table 1 Effect of R value on performance

[0086] The value of R Rank-1 mAP 0.5 69.5 59 0.6 69.2 58.8 0.7 70.8 60.4 0.8 71.9 61 0.9 70.3 60.5 1.0 69.2 58.8

[0087] Step 2.3, transform the original image X ori and the occlusion-enhanced image X occ Subtract pixel by pixel to get the mask image X of the occluded area mask , X mask The generation process can be expressed as formula (1):

[0088] X mask =X occ -X ori (1)

[0089] Among them, X mask Represents the generated mask image.

[0090] Step 3: Transform the original image X ori and the occlusion-enhanced image X occ At the same time, the preprocessed image X is obtained after image scaling and normalization. ori ' and Xocc ', described in detail as follows:

[0091] Step 3.1: Use bicubic interpolation to transform image X ori and X occ Scale to a fixed size: height 258, width 128, get the scaled image X ori_r and X occ_r ;

[0092] Step 3.2, for image X ori_r and X occ_r Normalization is performed. Specifically, the normalization process given a mean of (50, 50, 50) and a variance of std of (50, 50, 50) is as follows:

[0093] X for picture ori_r and X occ_r Each channel (red, green, blue) of each pixel is subtracted from the corresponding mean. Then, for the image X ori_r and X occ_r Each channel of each pixel is divided by the corresponding standard deviation (standard deviation is the square root of variance). The normalization process is shown in formula (2):

[0094]

[0095] Among them, c represents the RGB three channels (red, green, blue), X ori ' indicates X ori The normalized result, X occ ' indicates X occ As a result of normalization, the normalized image will have a pixel distribution with a mean close to 0 and a variance close to 1, which helps the model converge quickly and achieve better performance.

[0096] Step 4: Get the preprocessed original image X through step 3.2 ori ' and occlusion-enhanced image X occ ',like Figure 3 As shown, the image X ori ' and X occ 'The Vision Transformer backbone network is used to extract features (Backbone) to obtain global features and local features. Then, the local features are used to predict the occlusion mask (MaskPredict) through the mask predictor to obtain four regional masks {M 1 , M 2 , M 3 , M 4}, the predicted four regional masks are applied to the local feature sequence to obtain four local features that remove occlusion interference, and the four regional masks are maximized along the channel to obtain the foreground mask MF , the foreground mask is applied to the local feature sequence to obtain the global feature without occlusion interference, and then the global and local features without occlusion interference are input into the Transformer encoder for feature refinement to obtain more representative features, and then the global and local features are input into the feature context enhancement module for bidirectional enhancement to obtain a more robust feature representation, and then the enhanced global and local features are input into the classifier for classification, and finally multiple loss functions are used to jointly supervise learning to build a pedestrian re-identification model based on occlusion enhancement and mask learning (including VisionTransformer backbone network, mask predictor, Transformer encoder, feature context enhancement module and classifier). The specific steps are as follows:

[0097] Step 4.1, use VisionTransformer as the backbone network for feature extraction, and load the network weights pre-trained on the ImageNet dataset. Figure 4 As shown in the figure, the Vision Transformer backbone network consists of a patch embedding module and multiple Transformer encoders. ori ' and X occ As the input of the backbone network, the image is first divided into image blocks of equal size. The patch embedding module maps the image blocks to a higher embedding space, and then passes through the Transformer encoder for feature extraction. The following is a detailed description of each component and the entire forward processing process.

[0098] Step 4.1.1, given an input image X ori ' and X occ '. The Vision Transformer backbone network first takes the input image X ori ' and X occ 'Split into a series of fixed-size image blocks, each of which has a size of P×P (P refers to the width and height of the image block. In this method, P is set to 16). ori 'Get a series of image block sets through segmentation ( Represents any one of the image blocks, i = 1, 2, 3, ..., n, n = H ori *W ori / P 2 , H ori For X ori 'High, W ori For X ori 'width), X occ 'Get a series of image block sets through segmentation ( Represents any one of the image blocks, i = 1, 2, 3, ..., n, n = H occ *W / P 2 , H occ For X occ 'High, W occ For X occ 'width), and Flatten into a one-dimensional vector, and then map the vector to a higher-dimensional embedding space through a patch embedding module. This process can be expressed as formula (3):

[0099]

[0100] in, is an image block The mapped feature vector is Image Blocks The mapped feature vector, Flatten is the flattening operation, W e and b e are the weights and biases of the patch embedding module respectively.

[0101] Step 4.1.2: In order to preserve the position information, add the feature vector obtained in step 4.1.1 and Add positional encoding embedding E p_ori and E p_occ , while initializing the class tag CLS ori As the global feature representation of the original image, initialize the class label CLS occ As a global feature representation of the enhanced image, this process can be expressed as formula (4):

[0102]

[0103] Among them, E' ori express and position code E p_ori and the class tag CLS ori The combined eigenvector. E' occ express and position code E p_occ and the class tag CLS occ The combined feature vector.

[0104] Step 4.1.3, then the feature vector E' obtained in step 4.1.2 ori and E' occ Input Transformer encoder for feature extraction. Figure 4As shown in the figure, the Vision Transformer backbone network consists of multiple Transformer encoders (the present invention uses 12), and the Transformer encoder includes a multi-head self-attention mechanism (Multi-Head Self-Attention, MHSA) and a feed forward neural network (Feed Forward Network, FFN) and batch normalization (LayerNorm). E' ori The attention score MHSA (E' ori ), E' occ The attention score MHSA (E' occ ), then MHSA(E' ori ) is connected to the input E' through a residual connection ori Add together, MHSA(E' occ ) is connected to the input E' through a residual connection occ Add, and the result of the addition through the residual connection is finally normalized by LayerNorm to obtain E″ ori and E″ occ , then E″ ori and E″ OCC It will be processed by the feedforward neural network to obtain FFN(E″) ori ) and FFN(e″ OCC ), then FFN(E″ ori ) Using residual connections and E″ ori Add, FFN(E″ occ ) Using residual connections and E″ occ The result of the addition is then normalized by LayerNorm to obtain the final feature E″′ extracted by Transformer encoding. ori and E″′ occ , this process can be expressed as formula (5):

[0105]

[0106] Among them, MHSA represents the multi-head self-attention mechanism, and LayerNorm represents the layer normalized MHSA (E' ori ) represents E' ori The attention score obtained through MHSA processing, E″ ori Indicates E' ori Through the multi-head self-attention mechanism, residual connection and layer normalization processing, E″′ ori Represents R' ori The final output after feature extraction by Transformer encoder. MHSA(E'occ ) represents E' occ The attention score obtained through MHSA processing, E″ occ Indicates E' occ Through the multi-head self-attention mechanism, residual connection and layer normalization processing, E″′ occ Indicates E' occ The final output result after feature extraction by the Transformer encoder.

[0107] Step 4.2, after step 4.1 is completed, the extracted feature vector E″′ is obtained ori and E″′ occ . E″′ ori and E″′ occ is a sequence of n+1 feature vectors of dimension D (including class labels CLS ori / CLS occ and a sequence of n local features), extract the feature sequence E″′ ori and E″′ occ The class tag CLS ori and CLS occ As a global feature, the feature sequence E″′ ori and E″′ occ The feature vectors other than the class labels represent the local feature vector sequence, where E″′ ori The local eigenvector sequence of can be expressed as f p_ori , E″′ occ The local eigenvector sequence of can be expressed as f p_occ , f p_ori and f p_occ Prediction is performed through the Mask Predictor, such as Figure 5 As shown, the mask predictor includes a Relu activation function, a BN batch normalization layer, and a 1*1 convolution Conv, f p_ori and f p_occ First, it passes through the Relu activation function, then enters the BN batch normalization layer, then enters the 1*1 convolution Conv, and then generates four regional mask maps through SoftMax. The four regional masks correspond to different regions on the map. The partial region-based method can be complementary to the global-based method by matching the visible part in the case of severe occlusion. The mask prediction process is shown in formula (6):

[0108]

[0109] Among them, Reshape means changing the shape of the feature to the specified shape, Conv means 1*1 convolution, BN means batch normalization layer, ReLU means ReLU activation function, and SoftMax means SoftMax function. represents f p_ori Four region masks predicted by the mask predictor, represents f p_occ Four region masks predicted by the mask predictor.

[0110] Step 4.3, compared with global features, partial features can still be recognized by the unoccluded parts even when some parts are occluded. However, once the global features are severely occluded, the extraction of the overall features will be seriously affected, thereby reducing the recognition effect. Combining global features with local features can enhance the robustness of the model to occlusion:

[0111] Step 4.3.1, for the four region masks predicted in step 4.2 Taking the maximum value along the channel gives Also take the maximum value along the channel to get Then the class label CLS taken out in step 4.2 is ori and CLS occ As a global feature, Acts as foreground mask with CLS ori ,Will Acts as foreground mask with CLS occ , we get the global feature Z of occlusion suppression g_ori and Z g_occ , the above calculation process can be expressed by formula (7):

[0112]

[0113] Among them, TransformerBlock represents the Transformer encoder, cat represents the feature concatenation operation, max represents the maximum value operation, and [0] represents the extraction of class labels as global feature representation. and Represent the foreground masks of the original image and the enhanced image, respectively, Z g_ori and Z g_occ Represents the global feature of removing occlusion interference.

[0114] Step 4.3.2, mask the four regions obtained in step 4.2 Acting on the local eigenvector sequence f p_ori , and obtain four local features to remove occlusion interference Will The four region masks act on the local feature vector sequence f p_occ , and obtain four local features to remove occlusion interference Then the local features and global features CLS ori Splicing, local features and global features CLS occ Splicing, and then sending it to the Transformer encoder for feature refinement, so that the local features can better capture the global dependencies, and finally obtain the refined local features and The above calculation process can be expressed by formula (8):

[0115]

[0116] in, and Represent the i-th local features of the original image and the enhanced image respectively, and Respectively and The local features after occlusion suppression and feature refinement. cat represents the feature concatenation operation, where TransformerBlock represents the Transformer encoder, and [0] represents the class label extraction operation.

[0117] In step 4.4, many existing dual branches predict global features and local features respectively and then optimize them separately, but ignore the interaction between them. They also ignore the fact that pedestrians with different IDs may have highly similar local features (such as wearing the same style of clothes). The rich contextual information generated by the interaction helps to build a richer feature representation.

[0118] Based on this, the global feature Z obtained in step 4.3.1 is g_ori And the local features obtained in step 4.3.2 And the global feature Z obtained by enhancing the image g_occ and local features The feature context enhancement module is input into the feature context enhancement module for bidirectional enhancement to further obtain features with strong discrimination. Figure 6 As shown in the figure, it is composed of a nonlocal attention module. The nonlocal attention module consists of convolution and Softmax. First, the local features are enhanced by the global features, and then the global features are enhanced by the enhanced local features. The enhancement process is as follows:

[0119] Step 4.4.1, the global feature Z obtained from step 4.3.1 g_ori And the local features obtained in step 4.3.2 First use the global feature Z g_ori For local features Enhancement is completed in the non-local attention module to obtain enhanced local features This process can be expressed as formula (9):

[0120]

[0121] Among them, δ i (i=1,2,3) represents a 2D convolution of size 1*1, which plays the role of feature mapping. p_ori_1 i yes Through convolution δ 1 The result after mapping, f p_ori_2 ′ and f p_ori_2 ″ is Z g_ori Through convolution δ 2 and δ 3 The result after mapping is It means f p_ori_1 i and f p_ori_2 "Similarity vector, is the enhanced local feature, and the symbol ⊙ represents transposed multiplication.

[0122] The global feature Z obtained in step 4.3.1 g_occ And the local features obtained in step 4.3.2 First use the global feature Z g_occ For local features Enhancement is completed in the non-local attention module to obtain enhanced local features This process can be expressed as formula (10):

[0123]

[0124] Among them, δ i (i=1,2,3) represents a 2D convolution of size 1*1, which plays the role of feature mapping. p_occ_1 i yes Through convolution δ 1 The result after mapping, f p_occ_2 ′ and f p_occ_2 ″ is Z g_ori Through convolution δ 2 and δ 3 The result after mapping is It means f p_occ_1i and f p_pcc_2 "Similarity vector, is the enhanced local feature, and the symbol ⊙ represents transposed multiplication.

[0125] Step 4.4.2: The enhanced local features obtained in step 4.4.1 are For the global feature Z g_ori Enhance and supplement the fine-grained feature representation of the global feature to obtain the enhanced global feature Z g_ori ', this process can be expressed as formula (11):

[0126]

[0127] Among them, δ i (i=1,2,3) represents a 2D convolution of size 1*1, which plays the role of feature mapping. g_ori_1 It's Z g_ori Through convolution δ 1 The result after mapping is and yes Through convolution δ 2 and δ 3 The result after mapping is It means f g_ori_1 and The vector of similarity, Z g_ori ' is the enhanced local feature, and the symbol ⊙ represents transposed multiplication.

[0128] Similarly, the enhanced local features obtained in step 4.4.1 are For the global feature Z g_occ Enhance and supplement the fine-grained feature representation of the global feature to obtain the enhanced global feature Z g_occ ', this process can be expressed as formula (12):

[0129]

[0130] Among them, δ i (i=1,2,3) represents a 2D convolution of size 1*1, which plays the role of feature mapping. g_occ_1 It's Z g_occ Through convolution δ 1 The result after mapping is and yes Through convolution δ 2 and δ 3 The result after mapping is It means f g_occ_1 and The vector of similarity, Zg_occ ' is the enhanced local feature, and the symbol ⊙ represents transposed multiplication.

[0131] Step 4.5: The enhanced local features obtained in step 4.4.1 are and and the global feature Z g_ori ' and Z g_occ 'Classify through the classifier respectively to obtain the final classification result. The calculation process is shown in formula (13):

[0132]

[0133] Among them, linaer g and Represents the classifier, Score g_ori Represents the classification result of the global features of the original image and The classification result Score represents the local features of the original image g_occ Represents the classification result of the enhanced image global features, Represents the classification result of the enhanced image local features.

[0134] Step 5: Use the data preprocessed in step 3 to train the model built in step 4: Use the ID loss function, Triple loss function and Mask_DIV loss function to constrain the network, and then reversely propagate to update the parameters. After 170 rounds of training, 1 round here means training all the preprocessed images once, and finally obtain the trained pedestrian re-identification model based on occlusion perception and Transformer. The loss functions used in training are as follows:

[0135] ID loss function L id ID loss is the most common classification loss in person re-identification tasks, which is used to describe the distance between two probability distributions. In person re-identification, it is often used for identity-based classification tasks, enabling the model to distinguish different pedestrian identities. ID loss L id The formula is as follows (14):

[0136]

[0137] Among them, n is the number of samples in each batch training, N is the total number of batches, p(y i |x i ) is the input image x i and the class label y i , after SoftMax classification, x i Identified as y i The probability of the class.

[0138] Triplet loss L tri is a metric loss function that includes an anchor sample a i , a positive sample p i and a negative sample n i The goal of the triplet loss is to reduce the distance between the anchor point and the positive sample and increase the distance between the anchor point and the negative sample. If the distance between the anchor point and the positive sample minus the distance between the anchor point and the negative sample is less than the hyperparameter margin, the loss function will penalize the model parameters to force the model to adjust the feature representation so that the distance between the positive sample pairs is reduced and the distance between the negative sample pairs is increased. tri The calculation formula is as follows:

[0139]

[0140] Among them, d(a i ,p i ) represents the anchor point a i With positive sample p i The distance between i ,n i ) represents the anchor point a i With negative samples n i The distance between them, margin is a hyperparameter used to force the distance between positive and negative sample pairs to be greater than a fixed interval. M is the total number of triplets. In practical applications, the distance d is usually calculated using the Euclidean distance, as shown in formula (16):

[0141]

[0142] Among them, a i and b i They represent the i-th element in the vector, d is the Euclidean distance, and dim is the number of elements in the vector.

[0143] Mask_DIV loss function L mask Used to calculate the differences between multiple masks output by the pedestrian re-identification model. This loss function is designed to encourage the model to generate diverse mask representations, reduce the redundancy of predictions of different parts of the mask area, and improve the accuracy and diversity of mask predictions. Mask_DIV loss function L mask The formula is as follows (17):

[0144]

[0145] Where K is the number of masks, M i and M j are the outputs of the i-th and j-th masks respectively (0≤i, j≤4), and KL represents the KL divergence.

[0146] In summary, the joint learning loss function of the present invention is shown in formula (18):

[0147] Loss = L id +L tri +λL mask (18)

[0148] Among them, λ represents the weight of the Mask_DIV loss function in the total Loss.

[0149] Step 6: Put the image X in the validation set val After scaling and normalization, the processed image X is obtained val ', X val ' Input the model trained in step 5 and finally get the list of retrieved pedestrians.

[0150] Example 2

[0151] Based on Example 1, the specific process of step 6 is as follows:

[0152] Step 6.1: Use bicubic interpolation to transform image X val Scale to a fixed size: height 258, width 128, get the scaled image X val_r .

[0153] Step 6.2, for image X val_rr Normalize to get the preprocessed image X val ', specifically, the normalization process given a mean of (50,50,50) and a variance of (50,50,50) is as follows:

[0154] Step 6.2.1, subtract the mean: for image X val_r For each pixel, the corresponding mean is subtracted from each channel (red, green, blue).

[0155] Step 6.2.2, divide by the standard deviation: Then, for the image X val_r Each channel of each pixel is divided by the corresponding standard deviation (standard deviation is the square root of variance). The normalization process is shown in formula (19):

[0156]

[0157] Where c is the channel (red, green, blue), X val ' indicates X val The normalized result.

[0158] Step 6.3: image X processed in step 6.2 val ', put it into the model trained in step 5, and finally get the retrieved pedestrian list.

[0159] Example 3

[0160] The training parameters are set using the data in Table 2 below:

[0161] Table 2

[0162]

[0163]

[0164] The search and comparison results are as follows Figure 7 As shown, Figure 7 It contains two rows of images, the first row is the retrieval result of the comparison method, and the second row is the retrieval result of the present method. Each row in the figure includes 6 images, among which image A represents the pedestrian to be retrieved, B), C), D), E), F) are the retrieval results of the comparison method, and G), H), I), J), K) are the retrieval results of the method of the present invention. Among them, the images B)~F) and K) represent the wrong retrieval results, that is, the retrieved pedestrian and the pedestrian to be retrieved are not the same ID, and the images G)~J) represent the correct retrieval results, that is, the retrieved pedestrian and the pedestrian to be retrieved are the same ID. It can be seen that the method proposed in the present invention can effectively suppress the interference of the occluded area and perform matching according to the visible area, while the baseline network also takes the interference information of the occluded area into consideration and retrieves the wrong result. From Figure 7 The comparison results of the search list of the proposed method show that the proposed method can locate the visible area more accurately and retrieve the target pedestrian from the image library. The performance comparison between the proposed method and the existing method is shown in Table 3. The comparison indicators include two:

[0165] Rank-1 accuracy: indicates the probability of a correct match in the Top-1 search results of the query image. It reflects the accuracy of the model in the Top-1 results and is suitable for quickly evaluating whether the model can find the correct match.

[0166] mAP: The average of the mean precision of all query images, which reflects the matching performance of the model in the entire ranked list, provides a more comprehensive evaluation, especially in scenarios with multiple matching results.

[0167] It can be seen that the Rank-1 accuracy and mAP average accuracy of this method have reached advanced levels. Compared with the baseline network, the Rank-1 has increased by 8.7% and the mAP has increased by 4.9%, which greatly improves the accuracy of pedestrian re-identification under occlusion.

[0168] Table 3 below is a comparative experiment of the method of the present invention and other methods on the Occluded_DUKE dataset:

[0169] Table 3

[0170]

[0171]

[0172] This invention solves the domain bias and the problem of being unable to identify occluded pedestrians caused by using external models, improves the model's perception of occluded scenes, and effectively removes the interference of occluded areas on the re-identification network. The proposed mask diversity learning loss function DIV_LOSS can effectively reduce feature redundancy, allowing the model to learn more diverse, non-redundant and discriminative features.

Claims

1. A pedestrian re-identification method based on occlusion enhancement and mask learning, characterized by: The specific steps include: Step 1, obtain a data set, and divide the data set into a training set, a gallery set, and a query set, where the gallery set and the query set constitute a validation set; Step 2, perform occlusion enhancement on the images in the training set and generate a mask image; Step 3, the original image in the training set and the image generated after occlusion enhancement are simultaneously subjected to image scaling and normalization processing to obtain a preprocessed image; Step 4: Build a pedestrian re-identification model based on occlusion enhancement and mask learning; Step 5, using the image preprocessed in step 3 to train the model constructed in step 4; Step 6: After scaling and normalizing the images in the validation set, input them into the model trained in step 5 to finally obtain the retrieved pedestrian list.

2. The pedestrian re-identification method based on occlusion enhancement and mask learning according to claim 1, characterized in that: The specific process of step 2 is: Step 2.1, randomly sample the images in the Occluded-DukeMTMC query set to obtain image X, segment image X, manually select the occlusion blocks B formed by the occlusion area, and form an occlusion block image set S. The segmented occlusion block images are divided into two categories according to common occlusion and pedestrian occlusion, and the prior position information is added to the image file name respectively; Step 2.2: randomly generate a random number T between 0 and 1, and perform the training on the original image X ori Perform occlusion enhancement. If T<0.5, perform normal occlusion enhancement. If T ≥ 0.5, pedestrian occlusion enhancement is performed to obtain the enhanced occlusion image X occ ; Step 2.3, subtract the original image and the occlusion-enhanced image pixel by pixel to obtain the mask image X of the occluded area mask , X mask The generation process of is shown in the following formula (1): X mask =X occ -X ori (1) Among them, X msk Represents the generated mask image.

3. The pedestrian re-identification method based on occlusion enhancement and mask learning according to claim 2, characterized in that: The specific process of step 3 is as follows: Step 3.1: Use bicubic interpolation to interpolate image X ori and X occ Zoom in and get the zoomed image X ori_r and X occ_r ; Step 3.2, for image X ori_r and X occ_r Normalize them separately to get the normalized image X ori ′ and X occ ′.

4. The pedestrian re-identification method based on occlusion enhancement and mask learning according to claim 3, characterized in that: In the step 4, the pedestrian re-identification model based on occlusion enhancement and mask learning includes a Vision Transformer backbone network, a mask predictor, a Transformer encoder, a feature context enhancement module and a classifier; Image X ori ′ and X occ ′ After feature extraction through the Vision Transformer backbone network, global features and local features are obtained. Then, the local features are predicted by the mask predictor to obtain four regional masks. The predicted four regional masks are applied to the local feature sequence to obtain four local features that remove occlusion interference. The maximum value of the four regional masks along the channel is obtained to obtain the foreground mask M F , the foreground mask is applied to the local feature sequence to obtain the global feature without occlusion interference, and then the global and local features without occlusion interference are input into the Transfomer encoder for feature refinement to obtain more representative features, and then the global and local features are input into the feature context enhancement module for bidirectional enhancement to obtain a more robust feature representation, and then the enhanced global and local features are input into the classifier for classification.

5. The pedestrian re-identification method based on occlusion enhancement and mask learning according to claim 4, characterized in that: The specific process of step 4 is as follows: Step 4.1, use Vision Transformer as the backbone network for feature extraction to obtain the extracted feature vector E″′ ori and E″′ occ ; Step 4.2, extract the feature sequence E″′ respectively ori and E″′ occ The class tag CLS ori and CLS occ As a global feature, the feature sequence E″′ ori and E″′ occ The feature vectors other than the class labels represent the local feature vector sequence, where E″′ oci The local eigenvector sequence of p_ori , E″′ occ The local eigenvector sequence of p_occ , f p_ori and f p_occ The mask prediction is performed through the mask predictor. The mask prediction process is shown in formula (2): Among them, Reshape means changing the shape of the feature to the specified shape, Conv means 1*1 convolution, BN means batch normalization layer, ReLU means ReLU activation function, SoftMax means SoftMax function, represents f p_ori Four region masks predicted by the mask predictor, represents f p_occ Four region masks predicted by the mask predictor; Step 4.3, for the four region masks predicted in step 4.2 Taking the maximum value along the channel gives Also take the maximum value along the channel to get Then the class label CLS taken out in step 4.2 is ori and CLS occ As a global feature, Acts as foreground mask with CLS ori ,Will Acts as foreground mask with CLS occ , we get the global feature Z of occlusion suppression g_ori and Z g_occ , the above process is expressed by formula (3): Among them, TransformerBlock represents the Transformer encoder, cat represents the feature concatenation operation, max represents the maximum value operation, and [0] represents the extraction of class labels as global feature representation. and Represent the foreground masks of the original image and the enhanced image, respectively, Z g_ori and Z g_occ Represents the global features that remove occlusion interference; Step 4.4, mask the four regions Acting on the local eigenvector sequence f p_ori , and obtain four local features to remove occlusion interference Will The four region masks act on the local feature vector sequence f p_occ , and obtain four local features to remove occlusion interference Then the local features and global features CLS ori Splicing, local features and global features CLS occ Splicing, and then sending it to the Transformer encoder for feature refinement, and finally obtaining the refined local features and The above process is expressed by formula (4): in, and Represent the i-th local features of the original image and the enhanced image respectively, and Respectively and The local features after occlusion suppression and feature refinement, cat represents the feature concatenation operation, TransformerBlock represents the Transformer encoder, and [0] represents the class label extraction operation; Step 4.5: The global feature Z obtained in step 4.3 is g_ori And the local features obtained in step 4.4 And the global feature Z obtained by enhancing the image g_occ and local features The feature context enhancement modules are inputted separately for bidirectional enhancement, and then the enhanced local features are used to enhance the global features. Step 4.6, classify the enhanced local features and global features obtained in step 4.5 through classifiers respectively to obtain the final classification results.

6. The pedestrian re-identification method based on occlusion enhancement and mask learning according to claim 5, characterized in that: The specific process of step 4.1 is as follows: Step 4.1.1, given an input image X ori ′ and X occ ′, the Vision Transformer backbone network first transforms the input image X ori ′ and X occ ′ is divided into a series of fixed-size image blocks, each of which is of size P×P, X ori ′Get a series of image blocks through segmentation X occ ′Get a series of image blocks through segmentation Will and Flatten into a one-dimensional vector, and then map the vector to the embedding space through a patch embedding module. The above process is expressed as follows: in, is an image block The mapped feature vector is Image Blocks The mapped feature vector, Flatten is the flattening operation, W e and b e They are the weights and biases of the patch embedding module; Step 4.1.2, add the feature vector obtained in step 4.1.1 and Add positional encoding embedding E p_ori and E p_occ , while initializing the class tag CLS ori As the global feature representation of the original image, initialize the class label CLS occ As the global feature representation of the enhanced image, the above process is expressed by the following formula (6): Among them, E′ ori express and position code E p_ori and the class tag CLS ori The combined eigenvector; E′ occ express and position code E p_occ and the class tag CLS occ The feature vector after combination; Step 4.1.3, then the feature vector E′ obtained in step 4.1.2 ori and E′ occ Input Transformer encoder for feature extraction, E′ ori The attention score MHSA(E′) is obtained by processing the multi-head self-attention mechanism ori ), E′ occ The attention score MHSA(E′) is obtained by processing the multi-head self-attention mechanism occ ), MHSA(E′ ori ) is connected to the input E′ through a residual connection ori Add together, MHSA(E′ occ ) is connected to the input E′ through a residual connection occ Add, and the result of the addition through the residual connection is finally normalized by LayerNorm to obtain E″ ori and E″ occ , then E″ ori and E″ occ It will be processed by the feedforward neural network to obtain FFN(E″) ori ) and FFN(E″ occ ), FFN(E″ ori ) Using residual connections and E″ ori Add, FFN(E″ occ ) Using residual connections and E″ 2cc The result of the addition is then normalized by LayerNorm to obtain the final feature E″′ extracted by Transformer encoding. ori and E″′ occ , the above process is expressed by the following formula (7): Among them, MHSA represents the multi-head self-attention mechanism, and LayerNorm represents the layer normalized MHSA (E′ ori ) represents E′ ori The attention score obtained through MHSA processing, E″ ori Represents E′ ori Through the multi-head self-attention mechanism, residual connection and layer normalization processing, E″′ ori Represents E′ ori The final output result after feature extraction by Transformer encoder; MHSA(E′ occ ) represents E′ occ The attention score obtained through MHSA processing, E″ occ Represents E′ occ Through the multi-head self-attention mechanism, residual connection and layer normalization processing, E″′ occ Represents E′ occ The final output result after feature extraction by the Transformer encoder.

7. The pedestrian re-identification method based on occlusion enhancement and mask learning according to claim 6, characterized in that: The specific process of step 4.4 is as follows: Step 4.4.1, first use the global feature Z g_ori For local features Enhancement is completed in the non-local attention module to obtain enhanced local features The above process is expressed by the following formula (8): Among them, δ i represents a two-dimensional convolution of size 1*1, f p_ori_1 i yes The result after convolution δ1 mapping, f p_ori_2 ′ and f p_ori_2 ″ is Z g_ori The results after convolution δ2 and δ3 mapping, respectively, It means f p_ori_1 i and f p_ori_2 "Similarity vector, is the enhanced local feature, and the symbol ⊙ represents transposed multiplication; First use the global feature Z g_occ For local features Enhancement is completed in the non-local attention module to obtain enhanced local features This process is expressed by the following formula (9): Among them, f p_occ_1 i yes The result after convolution δ1 mapping, f p_occ_2 ′ and f p_occ_2 ″ is Z g_ori The results after convolution δ2 and δ3 mapping, respectively, It means f p_occ_1 i and f p_occ_2 "Similarity vector, is the enhanced local feature; Step 4.4.2: The enhanced local features obtained in step 4.4.1 are For the global feature Z g_ori Enhanced, the enhanced global feature Z is obtained g_ori ′, the above process is expressed by the following formula (10): Among them, f g_ori_1 It's Z g_ori The result after convolution δ1 mapping, and yes The results after convolution δ2 and δ3 mapping, respectively, It means f g_ori_1 and The vector of similarity, Z g_ori ′ is the enhanced local feature; Similarly, the enhanced local features obtained in step 4.4.1 are For the global feature Z g_occ Enhanced, the enhanced global feature Z is obtained g_occ ′, the above process is expressed by the following formula (11): Among them, f g_occ_1 It's Z g_occ The result after convolution δ1 mapping, and yes The results after convolution δ2 and δ3 mapping, respectively, It means f g_occ_1 and The vector of similarity, Z g_occ ′ is the enhanced local feature, and the symbol ⊙ represents transposed multiplication.

8. The pedestrian re-identification method based on occlusion enhancement and mask learning according to claim 7, characterized in that: During the training process of step 5, the loss function used is shown in the following formula (12): Loss=L id +L tri +λL mask (12) Among them, L id is the ID loss function, L tri is the triplet loss, L mask represents the Mask_DIV loss function, and λ represents the weight of the Mask_DIV loss function in the total Loss; Among them, ID loss L id The following formula (13) is used for calculation: Among them, n is the number of samples in each batch training, N is the total number of batches, p(y i |x i ) is the input image x i and the class label y i , after SoftMax classification, x i Identified as y i Probability of class; Triplet loss L tri The following formula (14) is used for calculation: Among them, d(a i ,p i ) represents the anchor point a i With positive sample p i The distance between i ,n i ) represents the anchor point a i The distance between the negative sample ni, margin is a hyperparameter, M is the total number of triplets; Mask_DIV loss function L mask The following formula (15) is used for calculation: Where K is the number of masks, M i and M j are the outputs of the i-th and j-th masks respectively, and KL represents the KL divergence.

9. The pedestrian re-identification method based on occlusion enhancement and mask learning according to claim 8, characterized in that: The specific process of step 6 is as follows: Step 6.1: Use bicubic interpolation to transform image X val Scaling is performed to obtain the scaled image X val_r ; Step 6.2, for image X val_rr Normalize to get the preprocessed image X val ′; Step 6.3: image X processed in step 6.2 val ′, put it into the model trained in step 5, and finally get the retrieved pedestrian list.

Images