Pedestrian re-identification method under occlusion

Abstract

The present application belongs to the technical field of pedestrian re-identification, and discloses a pedestrian re-identification method under a shielding condition.According to the feature that a target pedestrian appears in the center of a picture, a data augmentation method proposed by the present application randomly selects a non-central region of an original picture to replace with other pedestrian image blocks.On this basis, a non-target region suppression loss based on information flow is proposed for training a model to reduce the interference of a shielding region on final matching features.The pedestrian image block mixed data enhancement technology proposed by the present application can simulate various non-target pedestrian shielding conditions, expand the training data, and further enhance the robustness of the model to unknown shielding.The proposed non-target region suppression loss based on information flow reduces the information flow of a non-target region to final features, thereby reducing the noise introduced by shielding.The proposed data augmentation method and loss function are suitable for various network structures and do not increase the computational amount in the inference stage.

Description

Technical Field

[0001] This invention belongs to the field of pedestrian re-identification technology, specifically a pedestrian re-identification method under occlusion conditions based on pedestrian image patch hybrid data augmentation. Background Technology

[0002] Pedestrian re-identification refers to the task of matching target pedestrians among multiple non-overlapping cameras. It is a highly challenging computer vision task and is widely used as a core module in surveillance systems. In recent years, with the support of abundant training data provided by large-scale benchmark datasets and the powerful representation capabilities offered by deep learning, the accuracy of pedestrian re-identification has significantly improved. However, the occlusion problem caused by various obstacles such as vehicles, road signs, or other pedestrians has not been well resolved, and this situation is widespread in real-world applications.

[0003] The occlusion problem has two main challenges: 1) The features of occluded objects interfere with the features of the target pedestrian, introducing noise into the final features of each image. 2) Occlusion can cause missing parts of the target pedestrian's body, leading to feature misalignment during matching.

[0004] Some works employ local-to-local matching strategies to address the aforementioned problems. They first extract local features and capture the visibility of each part, then calculate the distance between matched local features and combine them into an overall distance for ranking based on visibility. Wang et al., in their 2022 AAAI conference paper "Pose-guided Feature Disentangling for Occluded Person Re-identification Based on Transformer," proposed using a pose-guided Transformer decoder structure to decouple local features and selectively match corresponding unoccluded parts. Somers et al., in their 2023 WACV conference paper "Body Part-Based Representation Learning for Occluded Person Re-Identification," used body part segmentation maps generated by an additional human parsing model as supervision information for local region locations, training the network to extract features from the corresponding regions. However, due to the insufficient occlusion diversity in current pedestrian datasets used for training, these methods struggle to cope with new occlusion scenarios, thus limiting their robustness.

[0005] Data augmentation is a method to expand training data and thus enhance the robustness of the model. Most augmentation methods used in occluded person re-identification can be seen as variations of the CutMix method, which randomly selects a region from the input image and replaces that region with another image patch. Zhuo et al., in their 2018 ICME paper "OccludedPerson Re-Identification," proposed using a random background image from the current image as the replacement image patch. Zhao et al., in their 2021 TIP journal paper "Incremental Generative Occlusion Adversarial Suppression Network for Person ReID," proposed an incremental approach to generating occlusion data from easy to difficult, progressively training the network to handle more severe occlusion situations. Most of these methods only consider simulating non-pedestrian occlusion, without considering the situation where multiple pedestrians appear simultaneously, i.e., non-target pedestrian occlusion. Wang et al., in their 2022 CVPR paper "Feature Erasing and Diffusion Network for Occluded Person Re-Identification," proposed simulating multi-pedestrian images by mixing features of non-target pedestrians into the original features. However, this method cannot guarantee the validity of the simulation, resulting in very limited performance improvement, and it also requires additional computational overhead. Summary of the Invention

[0006] To address the issue of low robustness to unknown occlusions caused by the low occlusion diversity in training data for occluded pedestrian re-identification models, this invention proposes a pedestrian image patch hybrid data augmentation method to simulate diverse multi-pedestrian scenarios. Based on the characteristic that target pedestrians appear in the center of the image, the proposed data augmentation method randomly selects a non-center region of the original image and replaces it with other pedestrian image patches. Furthermore, a non-target region suppression loss based on information flow is proposed for training the model to reduce the interference of occluded regions on the final matching features.

[0007] The technical solution of this invention:

[0008] A pedestrian re-identification method under occlusion conditions includes a pedestrian image patch hybrid data augmentation method and a non-target region suppression loss based on information flow, the steps of which are as follows:

[0009] Step 1: Prepare training pedestrian data;

[0010] Step 2: Construct a Transformer-based feature extraction model and a classification head module for training;

[0011] Step 3: Construct batch training data

[0012] Step 3-1: Use PK sampling to construct the batch raw data, that is, select P pedestrian identity IDs in each batch, and randomly sample K samples for each ID;

[0013] Step 3-2: Perform the following steps on all samples within the batch in sequence: scale them to the same size, fill the image with 0 values ​​around the edges, and crop them randomly to the same size;

[0014] Step 3-3: For each target sample x in the batch of original data a Another image is randomly selected from the training data batch as the occlusion sample x. b ;

[0015] Steps 3-4: Set the ratio r between the occlusion size and the original image size. o The size p of the mask block, and the ratio r of the mask. m ;

[0016] Steps 3-5: Select the offset ratio of the occluded sample relative to the target sample, and randomly select the horizontal offset ratio. and the vertical offset ratio r h =r o / r w ;in, It is a uniform distribution;

[0017] Steps 3-6: Calculate the absolute value of the horizontal offset w = W × r w And the absolute value of the vertical offset h = H × r h Where W and H are the width and height of the original input image, respectively;

[0018] Steps 3-7: Randomly select the offset direction in For Bernoulli distribution, d=0 indicates that the occlusion occurs in the lower left of the original image, and d=1 indicates that the occlusion occurs in the lower right.

[0019] Steps 3-8: Calculate the target sample x based on the offset direction and offset amount. a The obscured area R a , with the occluded image patch in the occluded sample x b The occlusion area R in b ;

[0020] Steps 3-9: Based on the mask block size p and mask ratio r m Generate a binary mask M;

[0021] Step 3-10: Replace target sample x a The occluded area in the image is the occluded image patch after masking:

[0022] x′ a ←x a

[0023] x′ a (R a )=x b (R b )*M+x a (R a )*(1-M);

[0024] Step 3-11: Calculate the obtained artificial occlusion data x′ a With target sample x a The batch training data is obtained by splicing the data together.

[0025] Step 4: Input the batch training data into the model built in Step 2 for forward propagation;

[0026] Step 5: Construct the loss function for training the person re-identification model, including the identity recognition loss function, the metric learning loss function, and the non-target region suppression loss based on information flow. The specific steps are as follows:

[0027] Step 5-1: Construct the Softmax loss function as the identity recognition loss function, which takes the class token features processed by Batchnorm as input;

[0028] Step 5-2: Construct a Triplet loss function as the metric learning loss function, which takes the final class token feature as input;

[0029] Step 5-3: Extract the self-attention matrix A from each Transformer layer. l ;

[0030]

[0031] Among them, Q l and K l N represents the query term and key of the output feature of the l-th layer, respectively. h This represents the number of heads in the multi-head attention, B is the number of images in the batch, and N is the number of tokens;

[0032] Step 5-4: For A l Along N h Taking the mean of the dimensions yields the overall attention A. l ←mean(A l );

[0033] Step 5-5: Use attention to expand the approximate information flow; from the lth j layer to li The attention unfolding of a layer can be represented as a recursive multiplication of attention layers:

[0034]

[0035]

[0036] Among them, A(l i ) indicates the lth i Self-attention of the layer, norm(*) denotes row normalization, and I denotes the identity matrix;

[0037] Steps 5-6: Construct a non-target region suppression loss function for training:

[0038]

[0039] in, This indicates the attention rollout from the occluded area to the final feature;

[0040] Step 6: Calculate the total loss function and update the model parameters using the backpropagation algorithm;

[0041] Step 7: Repeat steps 3 to 6 until the model converges;

[0042] Step 8: Inference phase, use the trained model to extract features of the query sample and all samples in the candidate set, calculate the Euclidean distance and sort them to obtain the final result.

[0043] The effects and benefits of this invention are as follows: the proposed pedestrian image patch hybrid data augmentation technique can simulate various non-target pedestrian occlusion situations, expand the training data, and thus enhance the robustness of the model to unknown occlusions; the proposed non-target region suppression loss based on information flow reduces the information flow from non-target regions to the final features, thereby reducing the noise introduced by occlusion; the proposed data augmentation method and loss function are applicable to various network structures and do not increase the computational load in the inference stage. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of a pedestrian image patch hybrid data augmentation method.

[0045] Figure 2 This is the overall framework diagram.

[0046] Figure 3 This is an attention visualization of our method versus the baseline method. Detailed Implementation

[0047] The specific embodiments of the present invention are further described below with reference to the accompanying drawings and technical solutions. The overall process is as follows: Figure 2 As shown.

[0048] Step 1: Prepare the training dataset. The training dataset can be an existing large-scale occluded pedestrian re-identification dataset, or a specific, self-constructed dataset. If it is a self-constructed dataset, you need to use bounding boxes to crop the pedestrian images and label the pedestrians with their identification tags;

[0049] Step 2: Build a deep learning model. Taking the simplest baseline model as an example, build a visual Transformer backbone network and load the pre-trained parameters; add a Batchnorm layer and a classification layer after the final output class token and initialize them.

[0050] Step 3: Construct training batch data. The specific steps are as follows:

[0051] Step 3-1: Use PK sampling to construct the batch raw data, that is, select P pedestrian identities (id) in each batch, and randomly sample K samples for each id;

[0052] Step 3-2: Perform the following steps on all samples in the batch in sequence: resize to the same size, pad with zero values ​​around the image, and crop randomly to the same size.

[0053] Step 3-3: For each sample x in the training data batch a Another image from the batch is randomly selected as the occlusion sample x. b ;

[0054] Steps 3-4: Set the ratio r between the occlusion size and the original image size. o The size p of the mask block, and the ratio r of the mask. m ;

[0055] Steps 3-5: Select the offset ratio of the occluded sample relative to the target sample, and randomly select the horizontal offset ratio. and the vertical offset ratio r h =r o / r w ,in It is a uniform distribution;

[0056] Steps 3-6: Calculate the absolute value of the horizontal offset w = W × r w And the absolute value of the vertical offset h = H × r h , where W and H are the width and height of the original input image, respectively;

[0057] Steps 3-7: Randomly select the offset direction in For Bernoulli distribution, d=0 indicates that the occlusion occurs in the lower left of the original image, and d=1 indicates that the occlusion occurs in the lower right.

[0058] Steps 3-8: Calculate the original image x based on the offset direction and offset amount. a The obscured area R a , with occluded image patches in x b Region R in b ;

[0059] Steps 3-9: Based on the mask block size p and mask ratio r m Generate a binary mask M;

[0060] Step 3-10: Replace the occluded areas in the original image with the masked occluded image blocks:

[0061] x′ a ←x a

[0062] x′ a (R a )=x b (R b )*M+x a (R a )*(1-M);

[0063] Step 3-11: Calculate the obtained artificial occlusion data x′ a Compared with the original data x a The batch training data is obtained by splicing the data together.

[0064] Step 4: Input the batch training data into the model built in Step 2 for forward propagation;

[0065] Step 5: Construct the pedestrian re-identification model training loss function, which includes an identity recognition loss function, a metric learning loss function, and a non-target region suppression loss based on information flow. The specific steps are as follows:

[0066] Step 5-1: Construct the Softmax loss function as the identity recognition loss function, which takes the class token that has undergone Batchnorm as input;

[0067] Step 5-2: Construct a Triplet loss function as the metric learning loss function, which takes the final class token feature as input;

[0068] Step 5-3: Extract the self-attention matrix A from each Transformer layer. l ;

[0069]

[0070] Q l and K lLet N represent the query and key for each feature (token) in layer l, respectively. h This represents the number of heads in multi-head attention, where B is the number of images in the batch and N is the number of tokens.

[0071] Step 5-4: For A l Along N h Taking the mean of the dimensions yields the overall attention A. l ←mean(A l );

[0072] Step 5-5: Approximate information flow using attention rollout. From step l... i layer to l i The attention unfolding of a layer can be represented as a recursive multiplication of attention layers:

[0073]

[0074]

[0075] Where A(l) i ) indicates the lth i Self-attention of the layer, norm(*) denotes row normalization, and I denotes the identity matrix;

[0076] Steps 5-6: Construct a non-target region suppression loss function for training:

[0077]

[0078] in This indicates the attention rollout from the occluded area to the final feature.

[0079] Step 6: Calculate the total loss function and update the model parameters using the backpropagation algorithm;

[0080] Step 7: Repeat steps 3 to 6 until the model converges;

[0081] Step 8: Inference phase, use the trained model to extract features (class tokens) of the query sample and all samples in the candidate set, calculate the Euclidean distance and sort them to obtain the final result.

Claims

1. A method for pedestrian re-identification under occlusion conditions, comprising a pedestrian image patch hybrid data augmentation method and a non-target region suppression loss based on information flow, characterized in that, The steps are as follows: Step 1: Prepare training pedestrian data; Step 2: Construct a Transformer-based feature extraction model and a classification head module for training; Step 3: Construct batch training data Step 3-1: Use PK sampling to construct the batch raw data, that is, select P pedestrian identity IDs in each batch, and randomly sample K samples for each ID; Step 3-2: Perform the following steps on all samples within the batch in sequence: scale them to the same size, fill the image with 0 values ​​around the edges, and crop them randomly to the same size; Step 3-3: For each target sample x in the batch of original data a Another image is randomly selected from the training data batch as the occlusion sample x. b ; Steps 3-4: Set the ratio r between the occlusion size and the original image size. o The size p of the mask block, and the ratio r of the mask. m ; Steps 3-5: Select the offset ratio of the occluded sample relative to the target sample, and randomly select the offset ratio in the horizontal direction. and the vertical offset ratio r h =r o / r w ;in, It is a uniform distribution; Steps 3-6: Calculate the absolute value of the horizontal offset w = W × r w And the absolute value of the vertical offset h = H × r h Where W and H are the width and height of the original input image, respectively; Steps 3-7: Randomly select the offset direction in For Bernoulli distribution, d=0 indicates that the occlusion occurs in the lower left of the original image, and d=1 indicates that the occlusion occurs in the lower right. Steps 3-8: Calculate the target sample x based on the offset direction and offset amount. a The obscured area R a , with the occluded image patch in the occluded sample x b The occlusion area R in b ; Steps 3-9: Based on the mask block size p and mask ratio r m Generate a binary mask M; Step 3-10: Replace target sample x a The occluded area in the image is the occluded image patch after masking: x′ a ←x a x′ a (R a )=x b (R b )*M+x a (R a )*(1-M); Step 3-11: Calculate the obtained artificial occlusion data x′ a With target sample x a The batch training data is obtained by splicing the data together. Step 4: Input the batch training data into the model built in Step 2 for forward propagation; Step 5: Construct the loss function for training the person re-identification model, including the identity recognition loss function, the metric learning loss function, and the non-target region suppression loss based on information flow. The specific steps are as follows: Step 5-1: Construct the Softmax loss function as the identity recognition loss function, which takes the class token features processed by Batchnorm as input; Step 5-2: Construct a Triplet loss function as the metric learning loss function, which takes the final class token feature as input; Step 5-3: Extract the self-attention matrix A from each Transformer layer. l ; Among them, Q l and K l N represents the query term and key of the output feature of the l-th layer, respectively. h This represents the number of heads in the multi-head attention, B is the number of images in the batch, and N is the number of tokens; Step 5-4: For A l Along N h Taking the mean of the dimensions yields the overall attention A. l ←mean(A l ); Step 5-5: Use attention to expand the approximate information flow; from the lth j layer to l i The attention unfolding of a layer can be represented as a recursive multiplication of attention layers: Among them, A(l i ) indicates the lth i Self-attention of the layer, norm(*) denotes row normalization, and I denotes the identity matrix; Steps 5-6: Construct a non-target region suppression loss function for training: in, This indicates the attention rollout from the occluded area to the final feature; Step 6: Calculate the total loss function and update the model parameters using the backpropagation algorithm; Step 7: Repeat steps 3 to 6 until the model converges; Step 8: Inference phase, use the trained model to extract features of the query sample and all samples in the candidate set, calculate the Euclidean distance and sort them to obtain the final result.

Images