Pedestrian re-identification method based on occlusion enhancement and mask learning

Abstract

This invention discloses a pedestrian re-identification method based on occlusion enhancement and mask learning, specifically including the following steps: Step 1, obtain a dataset and divide it into a training set, an image library set, and a query set, wherein the image library set and the query set constitute a validation set; Step 2, perform occlusion enhancement on the images in the training set and generate mask images; Step 3, simultaneously scale and normalize the original images and the images generated after occlusion enhancement in the training set to obtain preprocessed images; Step 4, construct a pedestrian re-identification model based on occlusion enhancement and mask learning; Step 5, train the model constructed in Step 4 using the preprocessed images from Step 3; Step 6, input the scaled and normalized images from the validation set into the model trained in Step 5 to finally obtain the retrieved list of pedestrians. This invention improves the robustness of pedestrian re-identification in occluded scenarios.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to a pedestrian re-identification method based on occlusion enhancement and mask learning. Background Technology

[0002] Person re-identification (ReID) is a computer vision task designed to identify the same person across different camera viewpoints. Its applications are wide-ranging, including video surveillance, intelligent security, and pedestrian tracking. However, in practical applications, pedestrian images can be subject to various interferences, the most challenging of which is occlusion. When a pedestrian is partially obscured by other objects (such as vehicles, trees, or other pedestrians), traditional person re-identification algorithms often struggle to accurately identify the target person. In this case, the person re-identification task is called occluded person re-identification.

[0003] Traditional pedestrian re-identification methods typically rely on hand-designed features and metric learning methods. For example, color histograms, texture features, and SIFT (Scale Invariant Feature Transform) are commonly used hand-designed features. These methods perform well in simple scenes, but their effectiveness is poor when dealing with complex environments (such as occlusion and changes in lighting).

[0004] Compared to traditional methods, deep learning methods, especially convolutional neural networks (CNNs), have demonstrated powerful performance in person re-identification.

[0005] In person re-identification, external models such as human body parsing and keypoint detection can assist in extracting human structural information and improve recognition accuracy. However, these methods also have some drawbacks. First, the introduction of external models increases system complexity and computational overhead, resulting in poor real-time performance. Second, external models often exhibit domain bias compared to person re-identification models; if the inference results of the external model are incorrect, it may lead to unreliable person re-identification results. Furthermore, these external models typically require pre-training and have poor adaptability to data distribution and scene changes, limiting their widespread application in practice. Summary of the Invention

[0006] The purpose of this invention is to provide a pedestrian re-identification method based on occlusion enhancement and mask learning. This method generates an accurate occlusion mask by subtracting the generated occlusion-enhanced image from the original image pixel by pixel, which serves as a supervision signal for the model and improves the robustness of pedestrian re-identification in occluded scenarios.

[0007] The technical solution adopted in this invention is a pedestrian re-identification method based on occlusion enhancement and mask learning, which specifically includes the following steps:

[0008] Step 1: Obtain the dataset and divide it into a training set, a graph set, and a query set, where the graph set and the query set constitute the validation set;

[0009] Step 2: Perform occlusion enhancement on the images in the training set and generate mask images;

[0010] Step 3: Simultaneously process the original image and the image generated after occlusion enhancement in the training set through image scaling and normalization to obtain the preprocessed image;

[0011] Step 4: Construct a pedestrian re-identification model based on occlusion enhancement and mask learning;

[0012] Step 5: Train the model constructed in Step 4 using the preprocessed image from Step 3;

[0013] Step 6: After scaling and normalizing the images in the validation set, input them into the model trained in Step 5 to finally obtain the list of retrieved pedestrians.

[0014] The invention is further characterized by:

[0015] The specific process of step 2 is as follows:

[0016] Step 2.1: Randomly sample the images in the Occluded-DukeMTMC query set to obtain image X. Segment image X, manually select the occlusion blocks B formed by the occlusion areas, and form an occlusion block image set S. Divide the segmented occlusion block images into two categories according to ordinary occlusion and pedestrian occlusion, and add prior location information to the image file names respectively.

[0017] Step 2.2: Randomly generate a random number T between 0 and 1, and apply it to the original image X sampled from the training set. ori Occlusion enhancement is performed. If T < 0.5, normal occlusion enhancement is performed; if T ≥ 0.5, pedestrian occlusion enhancement is performed, resulting in the enhanced image X. occ ;

[0018] Step 2.3: Subtract the original image and the occlusion enhancement image pixel by pixel to obtain the mask image X of the occlusion region. mask X mask The generation process is shown in the following formula (1):

[0019] X mask =X occ -X ori (1)

[0020] Among them, X mask This represents the generated mask image.

[0021] The specific process of step 3 is as follows:

[0022] Step 3.1, use bicubic interpolation to process image X. ori and X occ Scale the image to obtain the scaled image X. ori_r and X occ_r ;

[0023] Step 3.2, for image X ori_r and X occ_r Normalize each image separately to obtain the normalized image X. ori 'and X occ '.

[0024] In step 4, the pedestrian re-identification model based on occlusion enhancement and mask learning includes a Vision Transformer backbone network, a mask predictor, a Transformer encoder, a feature context enhancement module, and a classifier.

[0025] Image X ori 'and X occ Feature extraction is performed using the Vision Transformer backbone network to obtain global and local features. The local features are then used by a mask predictor to predict occlusion masks, resulting in four region masks. These four predicted region masks are applied to the local feature sequence to obtain four local features free from occlusion interference. Finally, the maximum value of each of the four region masks along its channel is used to obtain the foreground mask M. F The foreground mask is applied to the local feature sequence to obtain global features free from occlusion interference. Then, the global and local features free from occlusion interference are input into the Transformer encoder for feature refinement to obtain more representative features. Subsequently, the global and local features are input into the feature context enhancement module for bidirectional enhancement to obtain more robust feature representations. Finally, the enhanced global and local features are input into the classifier for classification.

[0026] The specific process of step 4 is as follows:

[0027] Step 4.1: Use VisionTransformer as the backbone network for feature extraction to obtain the extracted feature vector E”' ori and E”' occ ;

[0028] Step 4.2, extract the feature sequence E”' respectively ori and E”' occ Class marker CLS ori and CLS occ As a global feature, the feature sequence E”' ori and E”' occThe feature vectors other than the class label represent local feature vector sequences, where E”' ori The sequence of local feature vectors is represented as f p_ori ,E”' occ The sequence of local feature vectors is represented as f p_occ f p_ori and f p_occ The prediction is performed using a mask predictor, and the mask prediction process is shown in equation (2):

[0029]

[0030] Here, Reshape represents changing the shape of the feature to a specified shape, Conv represents a 1x1 convolution, BN represents a batch normalization layer, ReLU represents the ReLU activation function, and SoftMax represents the SoftMax function. f p_ori Four region masks predicted by the mask predictor f p_occ Four region masks predicted by a mask predictor;

[0031] Step 4.3, for the four region masks predicted in step 4.2 Take the maximum value along the channel to obtain We also obtain the maximum value along the channel. Then the class tag CLS retrieved in step 4.2 is used. ori and CLS occ As a global feature, As a foreground mask and CLS ori ,Will As a foreground mask and CLS occ The global feature Z of occlusion suppression is obtained. g_ori and Z g_occ The above process is represented by equation (3):

[0032]

[0033] Where TransformerBlock represents the Transformer encoder, cat represents the feature concatenation operation, max represents the maximum value operation, and [0] indicates that the class label is extracted as a global feature representation. and Z represents the foreground mask of the original image and the enhanced image, respectively. g_ori and Z g_occ This represents the global features after removing occlusion interference;

[0034] Step 4.4, mask the four regions. Acting on the local feature vector sequence f respectivelyp_ori Four local features were obtained after removing occlusion interference. Will The four region masks are applied to the local feature vector sequence f respectively. p_occ Four local features were obtained after removing occlusion interference. Then local features and global features CLS ori splicing, combining local features and global features CLS occ The concatenated features are then fed back into the Transformer encoder for feature refinement, ultimately yielding the refined local features. and The above process is represented by equation (4):

[0035]

[0036] in, and Let i and j represent the i-th local features of the original image and the enhanced image, respectively. and They represent and Local features after occlusion suppression and feature refinement, cat represents feature concatenation operation, where TransformerBlock represents Transformer encoder, and [0] represents class tag extraction operation;

[0037] Step 4.5, the global feature Z obtained in step 4.3 is... g_ori and the local features obtained in step 4.4 and the global feature Z obtained from the enhanced image g_occ and local features The feature context enhancement module is used for bidirectional enhancement.

[0038] Step 4.6: The enhanced local features and global features obtained in step 4.5 are classified by a classifier to obtain the final classification result.

[0039] The specific process of step 4.4 is as follows:

[0040] Step 4.4.1, first use the global feature Z g_ori For local features The enhancement is performed in the non-local attention module to obtain the enhanced local features. The above process is represented by the following formula (8):

[0041]

[0042] Where, δ i f represents a 2D convolution of size 1*1. p_ori_1 i yes f is obtained by mapping through convolution δ1. p_ori_2 'and f p_ori_2 "It is Z" g_ori The results after mapping by convolution δ2 and δ3 respectively It means f p_ori_1 i and f p_ori_2 "Similarity vectors, For the enhanced local features, the symbol ⊙ represents transpose multiplication;

[0043] First use global feature Z g_occ For local features The enhancement is performed in the non-local attention module to obtain the enhanced local features. The above process is represented by the following formula (9):

[0044]

[0045] Among them, f p_occ_1 i yes f is obtained by mapping through convolution δ1. p_pcc_2 'and f p_occ_2 "It is Z" g_ori The results after mapping by convolution δ2 and δ3 respectively It means f p_occ_1 i and f p_occ_2 "Similarity vectors, For enhanced local features;

[0046] Step 4.4.2, the enhanced local features obtained in step 4.4.1 For global feature Z g_ori Enhancement is performed to obtain the enhanced global feature Z. g_ori This process is represented by the following formula (10):

[0047]

[0048] Among them, f g_ori_1 It is Z g_ori The result after mapping by convolution δ1 and yes The results after mapping by convolution δ2 and δ3 respectively It means f g_ori_1 and Similarity vector, Zg_ori 'This refers to the enhanced local features;'

[0049] Similarly, the enhanced local features obtained in step 4.4.1 are... For global feature Z g_occ Enhancement is performed to obtain the enhanced global feature Z. g_occ The above process is represented by the following formula (11):

[0050]

[0051] Among them, f g_occ_1 It is Z g_occ The result after mapping by convolution δ1 and yes The results after mapping by convolution δ2 and δ3 respectively It means f g_occ_1 and Similarity vector, Z g_occ 'For the enhanced local features, the symbol ⊙ represents transpose multiplication.

[0052] In step 5 of the training process, the loss function used is shown in the following formula (12):

[0053] Loss = L id +L tri +λL mask (12)

[0054] Among them, L id For the ID loss function, L tri For triplet loss, L mask Let λ represent the Mask_DIV loss function, and let λ represent the weight of the Mask_DIV loss function in the total loss.

[0055] Among them, ID loss L id The following formula (13) is used for calculation:

[0056]

[0057] Where n is the number of training samples in each batch, N is the total number of batches, and p(y i |x i ) is the input image x i and category label y i After being categorized by SoftMax, x i Recognized as y i The probability of a class;

[0058] Triple loss L triThe following formula (14) is used for calculation:

[0059]

[0060] Wherein d(a i ,p i ) indicates anchor point a i Compared with positive sample p i The distance between them, d(a) i ,n i ) indicates anchor point a i With negative samples n i The distance between them, margin is a hyperparameter, and M is the total number of triples;

[0061] Mask_DIV loss function L mask The following formula (15) is used for calculation:

[0062]

[0063] Where K is the number of masks, M i and M j These are the outputs of the i-th and j-th masks, respectively, and KL represents the KL divergence.

[0064] The specific process of step 6 is as follows:

[0065] Step 6.1, use bicubic interpolation to interpolate image X. val Scale the image to obtain the scaled image X. val_r ;

[0066] Step 6.2, for image X val_rr Normalization is performed to obtain the preprocessed image X val ';

[0067] Step 6.3, process the image X from step 6.2. val ', put into the model trained in step 5, and finally get the list of retrieved pedestrians.

[0068] The beneficial effects of this invention are that the proposed method has the ability to perceive pedestrian occlusion and efficiently removes the adverse effects of occlusion regions on model feature extraction and accuracy. The occlusion simulation image enhancement method proposed in this invention can increase the occlusion data during training and generate masks for supervised learning of the model's mask prediction results, thereby achieving more accurate perception of occluded regions. By adding the Mask_DIV loss function, the model can learn more accurate and non-redundant mask regions. Furthermore, the added feature context enhancement module can bidirectionally enhance the representation capabilities of global and local features, improving model accuracy. Verification on public datasets demonstrates that the method of this invention has high metrics and excellent performance. Attached Figure Description

[0069] Figure 1 This is a flowchart illustrating the pedestrian re-identification method based on occlusion enhancement and mask learning of the present invention.

[0070] Figure 2 This is a schematic diagram of occlusion simulation in the pedestrian re-identification method based on occlusion enhancement and mask learning of the present invention;

[0071] Figure 3 This is a schematic diagram of the pedestrian re-identification model structure based on occlusion enhancement and mask learning constructed in the pedestrian re-identification method based on occlusion enhancement and mask learning of the present invention.

[0072] Figure 4 This is a schematic diagram of the Vision Transformer backbone network structure in the pedestrian re-identification method based on occlusion enhancement and mask learning of the present invention.

[0073] Figure 5 This is a structural diagram of the mask predictor in the pedestrian re-identification method based on occlusion enhancement and mask learning of the present invention;

[0074] Figure 6 This is a schematic diagram of the feature context enhancement module in the pedestrian re-identification method based on occlusion enhancement and mask learning of the present invention;

[0075] Figure 7 This is a visualization comparing the pedestrian re-identification method based on occlusion enhancement and mask learning of this invention with other methods. Detailed Implementation

[0076] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0077] Example 1

[0078] This invention relates to a pedestrian re-identification method based on occlusion enhancement and mask learning, such as... Figure 1 As shown, the specific steps include the following:

[0079] Step 1: The input data is the Occluded-DukeMTMC dataset, which is formed by selecting occlusion data from the DukeMTMC-reID dataset. It consists of three parts: a training set, an image library set, and a query set. The training set contains 15,618 images of 702 different pedestrians, the query set contains 2,210 images of 519 people, and the image library set contains 17,661 images of 1,110 different pedestrians. During the training phase, the training set is used for training. During the validation phase, the query set and the image library set form the validation set. The query set is used as the target pedestrian to be queried in the image library set.

[0080] Step 2: Perform occlusion enhancement on the image according to the occlusion enhancement strategy proposed in this invention. The specific process is as follows: Figure 2 As shown. First: Randomly sample the images in the query set of Occluded-DukeMTMC to obtain the original image X. Segment X, manually select the occlusion blocks B formed by the occlusion regions, and form an occlusion block image set S. Add prior location information to the file names of the occlusion block images, and then generate a random number T between 0 and 1. If T < 0.5, perform ordinary occlusion enhancement; if T ≥ 0.5, perform pedestrian occlusion enhancement. Then, take the original image X from the training set... ori Image X is generated after enhancement. occ X occ and X ori X is obtained by subtracting pixels one by one mask The detailed steps are described below:

[0081] Step 2.1: Randomly sample images X from the query set in Occluded-DukeMTMC to obtain images X. Segment X and manually select occlusion blocks B formed by occlusion regions to form an occlusion block image set S. Divide the segmented occlusion block images into two categories based on ordinary occlusion and pedestrian occlusion, and add prior position information to the image file names for each category. For ordinary occlusion, umbrellas usually occlude at the top of the image (top left or top right corner), and are usually aligned with the top of the entire image, so the occlusion block image file name for umbrellas is given the suffix "_1". For road signs and vehicle occlusion images, they are usually aligned with the bottom of the image (bottom left or bottom right corner), so the occlusion block image file name for road signs and vehicles is given the suffix "_2".

[0082] Step 2.2: Randomly generate a random number T between 0 and 1, and apply it to the original image X sampled from the training set. ori Perform occlusion enhancement. If T < 0.5, perform normal occlusion enhancement as shown in step 2.2.1. If T ≥ 0.5, perform pedestrian occlusion enhancement as shown in step 2.2.2.

[0083] Step 2.2.1: Normal Occlusion Enhancement: Randomly select a normal occlusion block image OP from the set of occlusion block images S obtained in Step 2.1. After reading the occlusion block image OP, read its name suffix. If the suffix is ​​"_1", then align the upper right corner of the occlusion block image OP with the original image X. ori Align the top right corner, then paste to generate the enhanced image X. occ .

[0084] Step 2.2.2, Pedestrian Occlusion Enhancement: Randomly select a pedestrian occlusion block image SP from the occlusion block image set S. First, based on the width H of the occlusion block SP... i and the original image X ori width HO Calculate the ratio R between the two. w If R w If the ratio is less than R, no action is taken; if R is less than R, no action is taken. w If the ratio of the occlusion block OP to the original image is greater than or equal to R, then the occlusion block OP is scaled to R times its original size, that is, the size of the occlusion block OP is scaled to 80% of its original size. In this invention, R is set to 0.8. The impact of the value of R on performance is shown in Table 1. As can be seen from Table 1, the performance is optimal when R is 0.8. The purpose of this is that in general, in real-world pedestrian occlusion situations, the occlusion area is not too large, and the target pedestrian will have a visible area. To prevent the occlusion block from completely obscuring the target pedestrian, it is scaled. Then, the lower left corner of the occlusion block image is compared with the original image X. ori Align the bottom left corner and paste to form an occlusion enhancement image X. occ .

[0085] Table 1 shows the impact of R values ​​on performance.

[0086] 0.5 69.5 59 0.6 69.2 58.8 0.7 70.8 60.4 0.8 71.9 61 0.9 70.3 60.5 1.0 69.2 58.8

[0087] Step 2.3, transform the original image X ori and occlusion enhancement image X occ Subtracting pixel by pixel yields the mask image X of the occluded area. mask X mask The generation process can be expressed as equation (1):

[0088] X mask =X occ -X ori (1)

[0089] Among them, X mask This represents the generated mask image.

[0090] Step 3, convert the original image X ori Image X with occlusion enhancement occ Simultaneously, the preprocessed image X is obtained after image scaling and normalization. ori 'and X occ The detailed description is as follows:

[0091] Step 3.1, use bicubic interpolation to interpolate image X. ori and X occ Scaling to a fixed size: 258 pixels high and 128 pixels wide, you get the scaled image X. ori_r and X occ_r ;

[0092] Step 3.2, for image X ori_r and X occ_rNormalization is performed, specifically, given a mean of (50, 50, 50) and a variance of (50, 50, 50), the normalization process is as follows:

[0093] For image X ori_r and X occ_r Subtract the corresponding mean from each channel (red, green, blue) of each pixel. Then, for image X... ori_r and X occ_r Each channel of each pixel is divided by its corresponding standard deviation (the standard deviation is the square root of the variance). The normalization process is shown in equation (2):

[0094]

[0095] Where 'c' represents the RGB three channels (red, green, blue), and 'X' represents the RGB three channels. ori 'represents X ori The result of normalization, X occ 'represents X occ The result of normalization is that the normalized image has a pixel distribution with a mean close to 0 and a variance close to 1, which helps the model converge quickly and achieve better performance.

[0096] Step 4: Obtain the preprocessed original image X through step 3.2. ori 'and occlusion enhancement image X occ ',like Figure 3 As shown, image X ori 'and X occ Feature extraction is performed using the Vision Transformer backbone network to obtain global and local features. The local features are then used by a mask predictor to predict occlusion masks, resulting in four region masks {M1, M2, M3, M4}. These four predicted region masks are applied to the local feature sequence to obtain four local features free from occlusion interference. Finally, the maximum value of each of the four region masks along its channels is used to obtain the foreground mask M. FThe process involves applying a foreground mask to a local feature sequence to obtain global features free from occlusion interference. These global and local features are then input into a Transformer encoder for feature refinement, resulting in more representative features. Next, the global and local features are input into a feature context enhancement module for bidirectional enhancement, yielding more robust feature representations. Finally, the enhanced global and local features are used to input into a classifier for classification. Finally, multiple loss functions are used for joint supervised learning to construct a pedestrian re-identification model based on occlusion enhancement and mask learning (including a VisionTransformer backbone network, mask predictor, Transformer encoder, feature context enhancement module, and classifier). The specific steps are as follows:

[0097] Step 4.1: Use VisionTransformer as the backbone network for feature extraction, and load the network weights pre-trained on the ImageNet dataset. For example... Figure 4 As shown, the Vision Transformer backbone network consists of a patch embedding module and multiple Transformer encoders. Image X ori 'and X occ As input to the backbone network, the image is first divided into image patches of equal size. The patch embedding module maps the image patches to a higher embedding space, and then the Transformer encoder performs feature extraction. The following is a detailed description of each component and the entire forward processing process.

[0098] Step 4.1.1, given the input image X ori 'and X occ The Vision Transformer backbone network first processes the input image X... ori 'and X occ The image is divided into a series of fixed-size blocks, each block being P×P (where P represents the width and height of the block; in this method, P is set to 16), X... ori A series of image patches are obtained through segmentation. ( Let i = 1, 2, 3, ..., n, and n = H. ori *W ori / P 2 H ori For X ori 'high, W ori For X ori (width of '), X occ A series of image patches are obtained through segmentation. ( Let i = 1, 2, 3, ..., n, and n = H. occ *W / P 2 H occ For X occ 'high, W occ For X occ 'width', will and The vector is flattened into a one-dimensional vector, and then mapped to a higher-dimensional embedding space through a patch embedding module. This process can be expressed as equation (3):

[0099]

[0100] in, It is an image block The mapped feature vector, Image Patch The mapped feature vector, Flatten is the flattening operation, W e and b e These are the weights and biases of the patch embedding module, respectively.

[0101] Step 4.1.2: In order to preserve positional information, the feature vector obtained in step 4.1.1 is... and Add position-encoded embedding E p_ori and E p_occ At the same time, initialize the class marker CLS. ori As a global feature representation of the original image, the initial class label CLS is used. occ As a global feature representation of the enhanced image, this process can be expressed as Equation (4):

[0102]

[0103] Among them, E' ori express and position code E p_ori and the class marker CLS ori The combined feature vector. E' occ express and position code E p_occ and the class marker CLS occ The combined feature vector.

[0104] Step 4.1.3, then the eigenvector E' obtained in step 4.1.2 is... ori and E' occ Input the Transformer encoder for feature extraction. For example... Figure 4As shown, the Vision Transformer backbone network consists of multiple Transformer encoders (12 in this invention). These encoders include a multi-head self-attention mechanism (MHSA), a feed-forward network (FFN), and layer normalization. ori Attention score MHSA(E') is obtained through multi-head self-attention mechanism. ori ), E' occ Attention score MHSA(E') is obtained through multi-head self-attention mechanism. occ ), then MHSA(E' ori ) connected to input E' via residual connection ori Added together, MHSA(E') occ ) connected to input E' via residual connection occ The summation, achieved through residual joins, is followed by layer normalization using LayerNorm to obtain E″. ori and E″ occ Then E″ ori and E″ OCC It will be processed by a feedforward neural network to obtain FFN(E″). ori ) and FFN(e″ OCC ), then FFN(E″ ori Using residual connections and E″ ori Add, FFN(E″) occ Using residual connections and E″ occ The sums are then normalized using LayerNorm, ultimately yielding the final feature E″′ extracted by the Transformer encoding. ori and E″′ occ This process can be expressed as equation (5):

[0105]

[0106] Wherein, MHSA represents multi-head self-attention mechanism, and LayerNorm represents layer-normalized MHSA(E'). ori ) represents E' ori The attention score obtained through MHSA processing, E″ ori E' ori Through multi-head self-attention mechanism, residual connection, and intermediate results processed by layer normalization, E″′ ori R' ori The final output after feature extraction by the Transformer encoder. MHSA(E'occ ) represents E' occ The attention score obtained through MHSA processing, E″ occ E' occ Through multi-head self-attention mechanism, residual connection, and intermediate results processed by layer normalization, E″′ occ E' occ The final output result after feature extraction by the Transformer encoder.

[0107] Step 4.2: After step 4.1 is completed, the extracted feature vector E″′ is obtained. ori and E″′ occ E″′ ori and E″′ occ A sequence of n+1 feature vectors of dimension D (including the class label CLS) ori / CLS occ (A sequence consisting of n local features), extract the feature sequence E″′ ori and E″′ occ Class marker CLS ori and CLS occ As a global feature, the feature sequence E″′ ori and E″′ occ The feature vectors other than the class label represent local feature vector sequences, where E″′ ori The sequence of local feature vectors can be represented as f p_ori ,E″′ occ The sequence of local feature vectors can be represented as f p_occ f p_ori and f p_occ Prediction is performed using a mask predictor, such as... Figure 5 As shown, the mask predictor includes a ReLU activation function, a BN batch normalization layer, and a 1x1 convolutional Conv,f p_ori and f p_occ First, the image passes through a ReLU activation function, then a BN batch normalization layer, followed by a 1*1 convolutional Conv layer. Finally, SoftMax generates four region mask images, each corresponding to a different region on the image. This partial region-based approach can match visible parts in cases of severe occlusion, complementing the global approach. The mask prediction process is shown in equation (6):

[0108]

[0109] Here, Reshape means changing the shape of the feature to a specified shape, Conv means 1*1 convolution, BN means batch normalization layer, ReLU means ReLU activation function, and SoftMax means SoftMax function. f p_ori Four region masks predicted by the mask predictor f p_occ Four region masks predicted by a mask predictor.

[0110] Step 4.3: Compared to global features, partial features can still be identified using the undisturbed parts even when some parts are occluded. However, for global features, severe occlusion significantly impacts overall feature extraction, thus reducing recognition accuracy. Combining global and local features can enhance the model's robustness to occlusion:

[0111] Step 4.3.1, for the four region masks predicted in step 4.2 Take the maximum value along the channel to obtain We also obtain the maximum value along the channel. Then the class tag CLS retrieved in step 4.2 is used. ori and CLS occ As a global feature, As a foreground mask and CLS ori ,Will As a foreground mask and CLS occ The global feature Z of occlusion suppression is obtained. g_ori and Z g_occ The above calculation process can be expressed by equation (7):

[0112]

[0113] Where TransformerBlock represents the Transformer encoder, cat represents the feature concatenation operation, max represents the maximum value operation, and [0] indicates that the class label is extracted as a global feature representation. and Z represents the foreground mask of the original image and the enhanced image, respectively. g_ori and Z g_occ This represents the global feature after removing occlusion interference.

[0114] Step 4.3.2, apply the four region masks obtained in Step 4.2. Acting on the local feature vector sequence f respectively p_ori Four local features were obtained after removing occlusion interference. Will The four region masks are applied to the local feature vector sequence f respectively. p_occ Four local features were obtained after removing occlusion interference. Then local features and global features CLS ori splicing, combining local features and global features CLS occ The concatenated features are then fed back into the Transformer encoder for feature refinement, allowing local features to better capture global dependencies, ultimately resulting in refined local features. and The above calculation process can be expressed by equation (8):

[0115]

[0116] in, and Let i and j represent the i-th local features of the original image and the enhanced image, respectively. and They represent and Local features after occlusion suppression and feature refinement. cat represents the feature concatenation operation, where TransformerBlock represents the Transformer encoder, and [0] represents the class label extraction operation.

[0117] Step 4.4: Many existing dual-branch methods predict global and local features separately and then optimize them separately, but ignore the interaction between them. They ignore the fact that pedestrians with different IDs may have highly similar local features (such as wearing the same style of clothing). The rich contextual information generated by the interaction helps to build richer feature representations.

[0118] Based on this, the global feature Z obtained in step 4.3.1 is... g_ori and the local features obtained in step 4.3.2 and the global feature Z obtained from the enhanced image g_occ and local features The feature context enhancement module performs bidirectional enhancement, inputting each feature separately to obtain more discriminative features. The feature context enhancement module is as follows: Figure 6 As shown, it consists of a nonlocal attention module. The nonlocal attention module comprises convolutions and softmax. It first enhances local features using global features, and then uses the enhanced local features to further enhance the global features. The enhancement process is as follows:

[0119] Step 4.4.1, the global feature Z obtained from step 4.3.1 g_ori and the local features obtained in step 4.3.2 First use global feature Z g_ori For local features The enhancement is performed in the non-local attention module to obtain the enhanced local features. This process can be represented by equation (9):

[0120]

[0121] Where, δ i (i = 1, 2, 3) represents a 1*1 two-dimensional convolution, which serves as a feature mapping, f p_ori_1 i yes f is obtained by mapping through convolution δ1. p_ori_2 ′ and f p_ori_2 "It is Z" g_ori The results after mapping by convolution δ2 and δ3 respectively It means f p_ori_1 i and f p_ori_2 "Similarity vectors, For the enhanced local features, the symbol ⊙ represents transpose multiplication.

[0122] The global feature Z obtained from step 4.3.1 g_occ and the local features obtained in step 4.3.2 First use global feature Z g_occ For local features The enhancement is performed in the non-local attention module to obtain the enhanced local features. This process can be expressed as equation (10):

[0123]

[0124] Where, δ i (i = 1, 2, 3) represents a 1*1 two-dimensional convolution, which serves as a feature mapping, f p_occ_1 i yes f is obtained by mapping through convolution δ1. p_occ_2 ′ and f p_occ_2 "It is Z" g_ori The results after mapping by convolution δ2 and δ3 respectively It means f p_occ_1 i and f p_pcc_2 "Similarity vectors, For the enhanced local features, the symbol ⊙ represents transpose multiplication.

[0125] Step 4.4.2, the enhanced local features obtained in step 4.4.1 For global feature Z g_ori Enhancement is performed to supplement the global feature with fine-grained feature representation, resulting in the enhanced global feature Z. g_ori This process can be expressed as equation (11):

[0126]

[0127] Where, δ i (i = 1, 2, 3) represents a 1*1 two-dimensional convolution, which serves as a feature mapping, f g_ori_1 It is Z g_ori The result after mapping by convolution δ1 and yes The results after mapping by convolution δ2 and δ3 respectively It means f g_ori_1 and Similarity vector, Z g_ori 'For the enhanced local features, the symbol ⊙ represents transpose multiplication.

[0128] Similarly, the enhanced local features obtained in step 4.4.1 are... For global feature Z g_occ Enhancement is performed to supplement the global feature with fine-grained feature representation, resulting in the enhanced global feature Z. g_occ This process can be expressed as equation (12):

[0129]

[0130] Where, δ i (i = 1, 2, 3) represents a 1*1 two-dimensional convolution, which serves as a feature mapping, f g_occ_1 It is Z g_occ The result after mapping by convolution δ1 and yes The results after mapping by convolution δ2 and δ3 respectively It means f g_occ_1 and Similarity vector, Z g_occ 'For the enhanced local features, the symbol ⊙ represents transpose multiplication.

[0131] Step 4.5, the enhanced local features obtained in step 4.4.1 and and global features Z g_ori 'and Z g_occThe classification is performed using classifiers to obtain the final classification result. The calculation process is shown in equation (13):

[0132]

[0133] Among them, linaer g and Score represents the classifier. g_ori Represents the classification results of global features of the original image and The classification result (Score) representing local features of the original image. g_occ The classification result represents the enhanced global features of the image. The classification result represents the enhanced local features of the image.

[0134] Step 5: Train the model built in Step 4 using the preprocessed data from Step 3. Constrain the network using the ID loss function, Triple loss function, and Mask_DIV loss function, then update the parameters via backpropagation. After 170 training rounds (one round refers to training all the preprocessed images once), a trained person re-identification model based on occlusion perception and Transformer is obtained. The specific loss functions used in the training are shown below:

[0135] ID loss function L id ID loss is the most common classification loss in person re-identification tasks, used to describe the distance between two probability distributions. In person re-identification, it is typically used for identity-based classification tasks, enabling the model to distinguish between different pedestrian identities. ID loss L id The formula is as shown in equation (14):

[0136]

[0137] Where n is the number of training samples in each batch, N is the total number of batches, and p(y i |x i ) is the input image x i and category label y i After being categorized by SoftMax, x i Recognized as y i The probability of a class.

[0138] Triple loss L tri It is a loss function that measures a loss function, which includes an anchor sample a. i A positive sample p i And a negative sample n iThe goal of triplet loss is to reduce the distance between the anchor point and positive samples, and increase the distance between the anchor point and negative samples. If the difference between the distance between the anchor point and positive samples and the distance between the anchor point and negative samples is less than the hyperparameter margin, the loss function will penalize the model parameters to prompt the model to adjust its feature representation, thereby reducing the distance between positive sample pairs and increasing the distance between negative sample pairs. The triplet loss L... tri The calculation formula is as shown in equation (15):

[0139]

[0140] Wherein d(a i ,p i ) indicates anchor point a i Compared with positive sample p i The distance between them. d(a) i ,n i ) indicates anchor point a i With negative samples n i The distance between positive and negative samples is a hyperparameter, where margin is used to force the distance between positive and negative sample pairs to be greater than a fixed interval. M is the total number of triples. In practical applications, the distance d is usually calculated using Euclidean distance, as shown in equation (16):

[0141]

[0142] Among them, a i and b i Let represent the i-th element in the vector, d be the Euclidean distance, and dim be the number of elements in the vector.

[0143] Mask_DIV loss function L mask This loss function is used to calculate the differences between multiple masks output by a person re-identification model. It aims to encourage the model to generate diverse mask representations, reduce redundancy in predictions of different mask regions, and improve the accuracy and diversity of mask predictions. The Mask_DIV loss function L... mask The formula is as follows (17):

[0144]

[0145] Where K is the number of masks, M i and M j These are the outputs of the i-th and j-th masks, respectively (0≤i, j≤4), and KL represents the KL divergence.

[0146] In summary, the joint learning loss function of this invention is shown in equation (18):

[0147] Loss = L id +L tri +λLmask (18)

[0148] Where λ represents the weight of the Mask_DIV loss function in the total loss.

[0149] Step 6, verify the image X in the set. val After scaling and normalization, the processed image X is obtained. val ', will X val Input the model trained in step 5 to obtain the final list of retrieved pedestrians.

[0150] Example 2

[0151] Based on Example 1, the specific process of step 6 is as follows:

[0152] Step 6.1, use bicubic interpolation to interpolate image X. val Scaling to a fixed size: 258 in height and 128 in width, yields the scaled image X. val_r .

[0153] Step 6.2, for image X val_rr Normalization is performed to obtain the preprocessed image X val Specifically, given a mean of (50, 50, 50) and a variance of (50, 50, 50), the normalization process is as follows:

[0154] Step 6.2.1, Subtract the mean: For image X val_r Subtract the corresponding mean from each channel (red, green, blue) of each pixel.

[0155] Step 6.2.2, divide by the standard deviation: Then, compare the X images. val_r Divide each channel of each pixel by the corresponding standard deviation (the standard deviation is the square root of the variance). The normalization process is shown in equation (19):

[0156]

[0157] Where c is the channel (red, green, blue), X val 'represents X val The result after normalization.

[0158] Step 6.3, process the image X from step 6.2. val ', put into the model trained in step 5, and finally get the list of retrieved pedestrians.

[0159] Example 3

[0160] The training parameters were set using the data in Table 2 below:

[0161] Table 2

[0162]

[0163]

[0164] Search and comparison results are as follows Figure 7 As shown, Figure 7 The image contains two rows of images. The first row shows the retrieval results of the comparison method, and the second row shows the retrieval results of the proposed method. Each row contains six images: image A represents the pedestrian to be retrieved; images B), C), D), E), and F) are the retrieval results of the comparison method; and images G), H), I), J), and K) are the retrieval results of the proposed method. Images B) to F) and K) represent incorrect retrieval results, i.e., the retrieved pedestrian and the pedestrian to be retrieved do not have the same ID; images G) to J) represent correct retrieval results, i.e., the retrieved pedestrian and the pedestrian to be retrieved have the same ID. It can be seen that the proposed method can effectively suppress interference from occluded areas and perform matching based on the visible area. However, the baseline network also considers the interference information from occluded areas, thus retrieving incorrect results. Figure 7 Based on the comparison results of the search lists, the method of this invention can more accurately locate the visible area and retrieve the target pedestrian from the image library. Table 3 shows a performance comparison between this method and existing methods, with two comparison metrics:

[0165] Rank-1 accuracy: Represents the probability of a correct match in the top-1 search results for a query image. It reflects the model's accuracy in the top-1 results and is suitable for quickly evaluating whether the model can find the correct match.

[0166] mAP: The average precision of all query images, reflecting the model's matching performance across the entire ranked list, providing a more comprehensive evaluation, especially effective in scenarios with multiple matching results.

[0167] It can be seen that the Rank-1 accuracy and mAP average accuracy of this method have reached the state-of-the-art level. Compared with the baseline network, the Rank-1 accuracy is improved by 8.7% and the mAP is improved by 4.9%, which greatly improves the accuracy of pedestrian re-identification under occlusion.

[0168] Table 3 below shows the comparative experiments of the method of this invention and other methods on the Occluded_DUKE dataset:

[0169] Table 3

[0170]

[0171]

[0172] This invention addresses the domain bias and inability to identify occluded pedestrians caused by using external models, improving the model's ability to perceive occluded scenes and effectively removing interference from occluded areas to the re-identification network. The proposed mask diversity learning loss function, DIV_LOSS, effectively reduces feature redundancy, enabling the model to learn more diverse, non-redundant, and discriminative features.

Claims

1. A pedestrian re-identification method based on occlusion enhancement and mask learning, characterized in that: Specifically, the steps include the following: Step 1: Obtain the dataset and divide it into a training set, a graph set, and a query set, where the graph set and the query set constitute the validation set; Step 2: Perform occlusion enhancement on the images in the training set and generate mask images; Step 3: Simultaneously process the original image and the image generated after occlusion enhancement in the training set through image scaling and normalization to obtain the preprocessed image; Step 4: Construct a pedestrian re-identification model based on occlusion enhancement and mask learning; Step 5: Train the model constructed in Step 4 using the preprocessed image from Step 3; Step 6: After scaling and normalizing the images in the validation set, input them into the model trained in Step 5 to finally obtain the list of retrieved pedestrians. The specific process of step 2 is as follows: Step 2.1: Randomly sample the images in the Occluded-DukeMTMC query set to obtain image X. Segment image X, manually select the occlusion blocks B formed by the occlusion areas, and form an occlusion block image set S. Divide the segmented occlusion block images into two categories according to ordinary occlusion and pedestrian occlusion, and add prior location information to the image file names respectively. Step 2.2, randomly generate a random number T between 0-1, for the original image X sampled in the training set ori occlusion enhancement, if T < 0.5, then normal occlusion enhancement is performed; If T ≥ 0.5, then pedestrian occlusion enhancement is performed to obtain the enhanced image. ; Step 2.3, subtract the original image and the occlusion enhanced image pixel by pixel to obtain the mask image X of the occlusion area mask , X mask The generation process is shown in the following formula (1): Among them, X mask This represents the generated mask image; The specific process of step 3 is as follows: Step 3.1, use bicubic interpolation to process image X. ori and X occ Scale the image to obtain the scaled image X. ori_r and X occ_r ; Step 3.2, for image X ori_r and X occ_r Normalize each image separately to obtain the normalized image. and ; In step 4, the pedestrian re-identification model based on occlusion enhancement and mask learning includes a Vision Transformer backbone network, a mask predictor, a Transformer encoder, a feature context enhancement module, and a classifier. Image and Feature extraction is performed using the Vision Transformer backbone network to obtain global and local features. The local features are then used by a mask predictor to predict occlusion masks, resulting in four region masks. These four predicted region masks are applied to the local feature sequence to obtain four local features free from occlusion interference. Finally, the foreground mask M is obtained by maximizing the values ​​of the four region masks along their respective channels. F The foreground mask is applied to the global feature sequence to obtain global features free from occlusion interference. Then, the global and local features free from occlusion interference are input into the Transformer encoder for feature refinement to obtain more representative features. The global and local features output from the Transformer encoder are then input into the feature context enhancement module for bidirectional enhancement. Finally, the enhanced global and local features are input into the classifier for classification.

2. The pedestrian re-identification method based on occlusion enhancement and mask learning according to claim 1, characterized in that: The specific process of step 4 is as follows: Step 4.1: Use Vision Transformer as the backbone network for feature extraction to obtain the extracted feature vector. and ; Step 4.2, extract the feature sequences respectively. and Class marker CLS ori and CLS occ As a global feature, the feature sequence and The feature vectors, excluding the class label, represent local feature vector sequences, where The sequence of local feature vectors is represented as f p_ori , The sequence of local feature vectors is represented as f p_occ f p_ori and f p_occ The prediction is performed using a mask predictor, and the mask prediction process is shown in equation (2): （2） Here, Reshape represents changing the shape of the feature to a specified shape, Conv represents a 1x1 convolution, BN represents a batch normalization layer, ReLU represents the ReLU activation function, and SoftMax represents the SoftMax function. f p_ori Four region masks predicted by the mask predictor f p_occ Four region masks predicted by a mask predictor; Step 4.3, for the four region masks predicted in step 4.2 Take the maximum value along the channel to obtain , We also obtain the maximum value along the channel. Then the class tag CLS retrieved in step 4.2 is used. ori and CLS occ As a global feature, As a foreground mask, it acts on CLS ori ,Will As a foreground mask, it acts on CLS occ The global feature Z of occlusion suppression is obtained. g_ori and Z g_occ The above process is represented by equation (3): （3） Where TransformerBlock represents the Transformer encoder, cat represents the feature concatenation operation, max represents the maximum value operation, and [0] indicates that the class label is extracted as a global feature representation. and Z represents the foreground mask of the original image and the enhanced image, respectively. g_ori and Z g_occ This represents the global features after removing occlusion interference; Step 4.4, mask the four regions. Acting on the local feature vector sequence f respectively p_ori Four local features were obtained after removing occlusion interference. ,Will Four region masks are applied to the local feature vector sequence f respectively. p_occ Four local features were obtained after removing occlusion interference. Then local features and global features CLS ori splicing, combining local features and global features CLS occ The concatenated features are then fed back into the Transformer encoder for feature refinement, ultimately yielding the refined local features. and The above process is represented by equation (4): （4） in, and Let i and j represent the i-th local features of the original image and the enhanced image, respectively. and They represent and Local features after occlusion suppression and feature refinement, cat represents feature concatenation operation, where TransformerBlock represents Transformer encoder, and [0] represents class tag extraction operation; Step 4.5, the global feature Z obtained in step 4.3 is... g_ori and the local features obtained in step 4.4 And the global feature Z obtained from the enhanced image g_occ and local features The feature context enhancement module is used to perform bidirectional enhancement, and then the enhanced local features are used to enhance the global features. Step 4.6: The enhanced local features and global features obtained in step 4.5 are classified by a classifier to obtain the final classification result.

3. The pedestrian re-identification method based on occlusion enhancement and mask learning according to claim 2, characterized in that: The specific process of step 4.1 is as follows: Step 4.1.1, given the input image and The Vision Transformer backbone network first processes the input image... and The image is divided into a series of fixed-size blocks, each block being P×P. A series of image patches are obtained through segmentation. , A series of image patches are obtained through segmentation. ,Will and Flatten it into a one-dimensional vector, and then map the vector to the embedding space through a patch embedding module. The above process is represented by the following formula (5): in, It is an image block The mapped feature vector, Image Patch The mapped feature vector, Flatten is the flattening operation, W e and b e These are the weights and biases of the patch embedding module, respectively; Step 4.1.2, input the feature vector obtained in step 4.1.

1. and Add position-encoded embedding E p_ori and E p_occ At the same time, initialize the class marker CLS. ori As a global feature representation of the original image, the initial class label CLS is used. occ As a global feature representation of the enhanced image, the above process is represented by the following formula (6): in, express and position code E p_ori and the class marker CLS ori The combined feature vector; express and position code E p_occ and the class marker CLS occ The combined feature vector; Step 4.1.3, then the feature vector obtained in step 4.1.2 is... and Input the Transformer encoder for feature extraction. Attention score MHSA is obtained through multi-head self-attention mechanism processing. ), Attention score MHSA is obtained through multi-head self-attention mechanism processing. ), MHSA (Through residual connection with input) Added together, MHSA ( (Through residual connection with input) The summation, achieved through residual joins, is finally normalized using LayerNorm. and Subsequently and It will be processed by a feedforward neural network to obtain FFN ( ) and FFN ( ), FFN( Using residual connections and Add, FFN( Using residual connections and The sums are then normalized using LayerNorm, ultimately yielding the final features extracted by the Transformer encoding. and The above process is represented by the following formula (7): （7） Where MHSA represents multi-head self-attention mechanism, and LayerNorm represents layer-normalized MHSA. )express Attention scores obtained through MHSA processing express Intermediate results processed using multi-head self-attention mechanism, residual connections, and layer normalization. express The final output after feature extraction by the Transformer encoder; MHSA ( )express Attention scores obtained through MHSA processing express Intermediate results processed using multi-head self-attention mechanism, residual connections, and layer normalization. express The final output result after feature extraction by the Transformer encoder.

4. The pedestrian re-identification method based on occlusion enhancement and mask learning according to claim 3, characterized in that: The specific process of step 4.5 is as follows: Step 4.5.1, first use the global feature Z g_ori For local features The enhancement is performed in the non-local attention module to obtain the enhanced local features. The above process is represented by the following formula (8): （8） Where, δ i This represents a 2D convolution of size 1*1. yes The result after mapping by convolution δ1 and It is Z g_ori The results after mapping by convolution δ2 and δ3 respectively It means and Similarity vectors For the enhanced local features, the symbol ⊙ represents transpose multiplication; First use global feature Z g_occ For local features The enhancement is performed in the non-local attention module to obtain the enhanced local features. This process is represented by the following formula (9): （9） in, yes The result after mapping by convolution δ1 and It is Z g_ori The results after mapping by convolution δ2 and δ3 respectively It means and Similarity vectors For enhanced local features; Step 4.5.2, the enhanced local features obtained in step 4.5.1 are... global features Enhancement is performed to obtain the enhanced global features. The above process is represented by the following formula (10): （10） in, It is Z g_ori The result after mapping by convolution δ1 and yes The results after mapping by convolution δ2 and δ3 respectively It means and Similarity vectors For enhanced local features; Similarly, the enhanced local features obtained in step 4.5.1 are... global features Enhancement is performed to obtain the enhanced global features. The above process is represented by the following formula (11): （11） in, yes The result after mapping by convolution δ1 and yes The results after mapping by convolution δ2 and δ3 respectively It means and Similarity vectors For the enhanced local features, the symbol ⊙ represents transpose multiplication.

5. The pedestrian re-identification method based on occlusion enhancement and mask learning according to claim 4, characterized in that: The loss function used in step 5 of the training process is shown in the following formula (12): （12） Among them, L id For ID loss function, For triple loss, Let λ represent the Mask_DIV loss function, and let λ represent the weight of the Mask_DIV loss function in the total loss. Among them, ID loss The following formula (13) is used for calculation: （13） Where n is the number of training samples in each batch, N is the total number of batches, and p(y i |x i ) is the input image x i and category label y i After being categorized by SoftMax, x i Recognized as y i The probability of a class; Triple loss The following formula (14) is used for calculation: （14） Wherein d(a i ,p i ) indicates anchor point a i Compared with positive sample p i The distance between them, d(a) i ,n i ) indicates anchor point a i With negative samples n i The distance between them, margin is a hyperparameter, and M is the total number of triples; Mask_DIV loss function The following formula (15) is used for calculation: （15） Where K is the number of masks, M i and M j These are the outputs of the i-th and j-th masks, respectively, and KL represents the KL divergence.

6. The pedestrian re-identification method based on occlusion enhancement and mask learning according to claim 5, characterized in that: The specific process of step 6 is as follows: Step 6.1, use bicubic interpolation to transform the image. Scale the image to obtain the scaled image. ; Step 6.2, for the image r Normalization is performed to obtain the preprocessed image. ; Step 6.3: Process the image from step 6.

2. The data is fed into the model trained in step 5, and the final list of retrieved pedestrians is obtained.

Images