Occlusion-based pedestrian re-identification method based on pose estimation and Transformer
By combining pose estimation and Transformer, pedestrian images are divided into key point regions and feature distances are calculated using weighted averages. This addresses the shortcomings of feature extraction and matching in occluded pedestrian re-identification and improves the re-identification accuracy in occluded scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XUZHOU YUNBIANDUAN INTELLIGENT TECH CO LTD
- Filing Date
- 2022-11-08
- Publication Date
- 2026-07-03
AI Technical Summary
Existing methods for re-identifying occluded pedestrians struggle to effectively extract robust local features and perform matching in occluded scenarios, and fail to effectively combine human pose estimation information with self-attention mechanisms, resulting in insufficient re-identification accuracy.
Pedestrian images are divided into human keypoint regions and background regions using a pose estimation network. Features are extracted using keypoint tokens, and global and local feature distances are calculated by weighting. Combined with the multi-head self-attention mechanism of Vision Transformer, the feature extraction network is optimized to improve matching accuracy.
It effectively improves the accuracy of pedestrian re-identification in occluded scenarios and enhances the robustness of feature extraction and matching accuracy under occluded conditions.
Smart Images
Figure CN115841682B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of pedestrian re-identification technology, and in particular to an occluded pedestrian re-identification method based on pose estimation and Transformer. Background Technology
[0002] Pedestrian re-identification technology aims to associate specific pedestrians with different scenes and perspectives from different cameras. In real-world surveillance scenarios, pedestrians are inevitably obscured, posing a significant challenge to feature extraction for re-identification. When occlusion exists, since some individuals are not visible, extracting robust local features and utilizing these features for matching is crucial for solving the problem of pedestrian re-identification in occlusion.
[0003] Existing methods for occluded pedestrian re-identification can be divided into manual segmentation and methods based on human semantic models. The former divides pedestrian images into multiple pre-defined local regions by horizontal or block segmentation, and extracts local features within these regions. However, manual segmentation methods cannot align human body parts and introduce a large amount of background noise. The latter is often based on human semantic models. Existing methods often first use a re-identification model and a human semantic model to extract re-identification features and human semantic information respectively, and then use the human semantic information to post-process the re-identification features to obtain local features. These methods only process the re-identification features output by the re-identification model and do not incorporate human semantic information into the extraction process of re-identification features.
[0004] Most existing methods for re-identifying occluded pedestrians are based on convolutional neural networks (CNNs). With the development of deep learning technology, the global multi-head self-attention mechanism in Vision Transformer networks has overcome the limitations of traditional CNNs in capturing global features and addressed the information loss caused by downsampling. Existing Vision Transformer-based methods for re-identifying occluded pedestrians improve the robustness of feature extraction by refining the self-attention mechanism, but they do not consider combining human pose estimation information with the self-attention mechanism to address local feature extraction and human body part alignment issues. Summary of the Invention
[0005] The purpose of this invention is to provide a pedestrian re-identification method based on pose estimation and Transformer. Pose estimation is used to extract human pose information, and the human body region is divided using the pose information. Keypoint tokens are introduced to extract keypoint features of pedestrians respectively. The final pedestrian distance is obtained by weighting the keypoint distance using keypoint confidence, thereby improving the accuracy of pedestrian re-identification algorithm in occluded scenarios.
[0006] This invention proposes an occluded person re-identification method based on pose estimation and Transformer, comprising the following steps:
[0007] The pose estimation network extracts the pose estimation information of pedestrians in the pedestrian image, obtains the highest confidence in each human key point region in the pedestrian image, and divides the pedestrian image into K human key point regions and 1 background region.
[0008] Construct a feature extraction network based on Vision Transformer to extract the global human feature matrix f of pedestrian images. g Human body key point feature matrix
[0009] Matching is performed on global human body features and keypoint human body features. The distance between global human body features and the distance between K keypoint human body features are calculated, and a weighted average is calculated between the global human body feature distance and the keypoint feature distance. The formula for calculating the distance is as follows:
[0010]
[0011] In the above formula, d() is the cosine distance, and f a,g and f b,g Let A and B be the global human feature matrices of a given pedestrian image a and pedestrian image b, respectively. and Let be the feature matrices of the i-th human body key points in given pedestrian image a and pedestrian image b, respectively. and Let λ be the highest confidence score within the i-th keypoint region of given pedestrian image a and pedestrian image b, respectively. g and λ k This is the weighting factor.
[0012] In some implementations, the method for dividing the region includes the following steps:
[0013] Set a confidence threshold δ.
[0014] Pose estimation network is used to extract pose estimation information of pedestrians. For each point in the space corresponding to the pedestrian image, the confidence level relative to K human key points is obtained.
[0015] Extract the highest confidence score from the K confidence scores corresponding to each point, and determine whether the highest confidence score is greater than the threshold δ. If it is greater, the point is assigned to the human key point region corresponding to the current highest confidence score; otherwise, the point is assigned to the background region.
[0016] In some implementations, K is set to 14, and the 14 key human body points include the left and right elbows, left and right wrists, left and right shoulders, head, neck, left and right ankles, left and right knees, and left and right hips.
[0017] In some implementations, the global human feature matrix f of the pedestrian image is extracted. g Human body key point feature matrix The process is as follows:
[0018] The pedestrian image is divided into N blocks of fixed size. N block features are obtained through linear mapping. The region segmentation method is applied to classify each block feature into the corresponding human key point region.
[0019] Create one global token and K keypoint tokens. Concatenate the N block features of the one global token, K keypoint tokens, and the keypoint tokens to obtain an input sequence of length 1+K+N. Feed the input sequence into the encoder of the Vision Transformer.
[0020] The multi-head self-attention calculation formula is constructed on the encoder of the Vision Transformer, and the formula is as follows:
[0021]
[0022] In the above formula, Q is the query vector, K is the key vector, V is the value vector, D is the dimension of the query vector, and Softmax() is the softmax function.
[0023] The global token and all block features are mapped to the spaces corresponding to Q, K, and V, and multi-head attention is performed to obtain the human global feature matrix f. g ;
[0024] The keypoint tokens and the block features within the keypoint regions corresponding to the keypoint tokens are mapped to the Q, K, and V spaces, and multi-head attention is performed to obtain the human keypoint feature matrix.
[0025] In some implementations, the Vision Transformer-based feature extraction network applies a triplet loss function and a classification loss function optimization model. The parameters of the Vision Transformer-based feature extraction network are adjusted based on the triplet loss function and the classification loss function. The calculation formulas for the loss functions in the triplet loss function and classification loss function optimization model include:
[0026] Formula for calculating triplet loss of human keypoint features:
[0027] The formula for calculating the classification loss of human body keypoint features:
[0028] The triplet loss function and the classification loss function optimize the final loss function of the model:
[0029] In the above formula
[0030] d(a,p) and d(a,n) are the distances between the features of the anchor sample and the positive sample, respectively, and m is a parameter;
[0031] N is the number of sample categories, Z i This represents the output classification probability;
[0032] and These are the triplet loss and classification loss for global human body features, respectively. and Let λ' be the triplet loss and classification loss for the keypoint features of the i-th person. g and λ' k This is the weighting factor.
[0033] In some implementations, the λ' g and λ' k The values are 0.5 and .
[0034] In some implementations, the λ g and λ k 1 and
[0035] A processing apparatus includes a processor and a storage module, the storage module being used to store a program, and the processor being used to load and execute the program to implement an occluded pedestrian re-identification method based on pose estimation and Transformer.
[0036] The occlusion-based person re-identification method described in this invention has the following advantages:
[0037] 1. Use a pose estimation network to obtain pedestrian pose estimation information, and divide the pedestrian image into human key point regions and background regions based on the pose estimation information, effectively separating the visible human key point regions from the background and occlusion.
[0038] 2. Using keypoint tokens to extract keypoint features can yield robust and discriminative features;
[0039] 3. Effectively improves the accuracy of pedestrian re-identification in occluded scenarios. Attached Figure Description
[0040] Figure 1 This is a flowchart of an occluded pedestrian re-identification method based on pose estimation and Transformer in some embodiments of the present invention;
[0041] Figure 2 The output image shows the results of re-identification retrieval using an occlusion-based pedestrian re-identification method based on pose estimation and Transformer in some embodiments of the present invention. Detailed Implementation
[0042] A confidence threshold δ is pre-set in the pose estimation network HR-Net, and the region to which a point belongs is determined by the threshold δ.
[0043] Combination Figure 1 The present embodiment proposes an occluded person re-identification method based on pose estimation and Transformer, including the following steps:
[0044] S1. Given pedestrian image a and pedestrian image b;
[0045] S2. Use the HR-Net pose estimation network to extract pedestrian pose estimation information, which includes the confidence scores of K human keypoints. Based on the confidence scores, pedestrian image a and pedestrian image b are spatially divided into K keypoint regions and 1 background region. The specific division process includes the following steps:
[0046] S21. Use the pose estimation network HR-Net to extract the pose estimation information of pedestrians and obtain the confidence of each point in the space corresponding to pedestrian image a and pedestrian image b relative to K human key points.
[0047] S22. Extract the highest confidence score from the K confidence scores corresponding to each point, and determine whether the highest confidence score is greater than the threshold δ. If it is greater, then classify the point into the human key point region corresponding to the current highest confidence score; otherwise, classify the point into the background region.
[0048] It should be noted that,
[0049] The confidence level can be obtained directly using existing technologies, so it will not be elaborated here.
[0050] S3. Construct a feature extraction network based on Vision Transformer to extract the global human feature matrix f of pedestrian image a and pedestrian image b. g Human body key point feature matrix The specific process includes the following steps:
[0051] S31. Divide pedestrian image a and pedestrian image b into N blocks of fixed size. Obtain N block features through linear mapping. Apply the region division method (i.e., apply the content described in S2, S21 and S22 above) to assign each block feature to the corresponding human key point region. For example, resize the pedestrian image to (256, 128), where 256 and 128 represent the number of pixels. Divide the image into 128 blocks of fixed size. Obtain 128 block features through linear mapping. Then, according to the region division method, assign each block feature to the corresponding region.
[0052] S32. Create 1 global token and K keypoint tokens. Concatenate the 1 global token, K keypoint tokens, and N block features to obtain an input sequence of length 1+K+N. Feed the input sequence into the encoder of Vision Transformer.
[0053] S33. Construct a multi-head self-attention calculation formula on the encoder of the Vision Transformer. The formula is as follows:
[0054]
[0055] In the above formula, Q is the query vector, K is the key vector, V is the value vector, D is the dimension of the query vector, and Softmax() is the softmax function.
[0056] S34. Map the global token and all block features to the spaces corresponding to Q, K, and V, perform multi-head attention calculation, and the output cls token is the human global feature matrix f. g ;
[0057] S35. Map the keypoint token and the block features within the keypoint region corresponding to the keypoint token to the Q, K, and V spaces, perform multi-head attention calculation, and the output cls token is the human body keypoint feature matrix.
[0058] S4. Calculate the global feature distance of the human body and the feature distance of K human body key points. Then, perform a weighted average of the global feature distance and the key point feature distance to obtain the feature distance between pedestrian images. Compare the feature distance between pedestrian images with a pre-set feature distance threshold to determine whether the global features and human body key point features of two pedestrian images match. Assuming a given pedestrian image a and pedestrian image b, the formula for calculating the feature distance between pedestrian image a and pedestrian image b is:
[0059]
[0060] In the above formula, d() is the cosine distance, and fa,g and f b,g These are the global human feature matrices for pedestrian image a and pedestrian image b in the pedestrian image group, respectively. and Let be the feature matrices of the i-th human body key points in pedestrian image a and pedestrian image b, respectively. and Let λ be the highest confidence score within the i-th keypoint region of pedestrian image a and pedestrian image b, respectively. g and λ k This is the weighting factor.
[0061] In S3, the Vision Transformer-based feature extraction network applies a triplet loss function and a classification loss function optimization model. The parameters of the Vision Transformer-based feature extraction network are adjusted based on these two loss functions to continuously optimize the extracted global human feature matrix and human keypoint feature matrix. The calculation formulas for the loss functions in the triplet loss function and classification loss function optimization model include:
[0062] Formula for calculating triplet loss of human keypoint features:
[0063] The formula for calculating the classification loss of human body keypoint features:
[0064] The triplet loss function and the classification loss function optimize the final loss function of the model:
[0065] In the above formula
[0066] d(a,p) and d(a,n) are the distances between the features of the anchor sample and the positive sample, respectively, and m is the parameter of the feature extraction network based on VisionTransformer;
[0067] N is the number of sample categories, Z i This represents the output classification probability;
[0068] and These are the triplet loss and classification loss for global human body features, respectively. and Let λ' be the triplet loss and classification loss for the keypoint features of the i-th person. g and λ' k This is the weighting factor.
[0069] The above triplet loss function and classification loss function optimization model adopt the commonly used pk sampling and hardmining deep learning hard sample mining techniques. For samples in a batch (each batch includes p people, each person has k′ images, a total of p*k′ samples, the values of p and k′ can be set according to requirements), when making anchor samples for each image, the image with the closest different ID is selected as the negative sample, and the image with the same ID that is farthest away is selected as the positive sample. Different images of the same person are set with the same ID, and images of different people are set with different IDs.
[0070] In some specific implementation methods,
[0071] The number of key points on the human body, K, is 14. The 14 key points on the human body include the left and right elbows, left and right wrists, left and right shoulders, head, neck, left and right ankles, left and right knees, and left and right hips.
[0072] In some specific implementations, the confidence threshold δ in the attitude estimation network HR-Net is pre-set to 0.5.
[0073] In some specific implementations, the triplet loss function and the classification loss function optimize the weight factor λ' of the model. g and λ' k The value can be set to 0.5 and , respectively.
[0074] The dimension D of the query vector is 768;
[0075] The weighting factor λ when weighting the global feature distance of the human body and the feature distance of key points g and λ k 1 and
[0076] For example, the occlusion-based pedestrian re-identification method based on pose estimation and Transformer can be applied to a database containing multiple images for re-identification retrieval. One image or a specified image from the database is input as the baseline pedestrian image. The occlusion-based pedestrian re-identification method based on pose estimation and Transformer is applied to re-identify the images in the database against the baseline pedestrian image, obtaining images that match the baseline pedestrian image. The 10 images with the closest feature distance to the baseline pedestrian image among the successfully matched images are then output, forming... Figure 2 The results are shown.
[0077] Comparative recognition experiments were conducted on the Occlude-Duke, Occluded-REID, and Partial-REID datasets using the Baseline method (existing benchmark methods for person re-identification, such as the results obtained using TransReID: Transformer-based Object Re-Identification published at ICCV 2021 with the standard Vision Transformer Base as the backbone network) and Ours method (the occluded person re-identification method based on pose estimation and Transformer described in this invention). The experimental data are shown in Table 1 below.
[0078] Table 1 shows the experiments on the Occlude-Duke, Occluded-REID, and Partial-REID datasets.
[0079]
[0080] In summary, the method proposed in this invention achieves Rank-1 values of 66.1, 73.4, and 75.0 and mAP values of 57.1, 69.4, and 72.1 on the Occlude-Duke, Occluded-REID, and Partial-REID datasets for occluded person re-identification, respectively, effectively improving the accuracy of occluded person re-identification.
[0081] One specific implementation proposes a processing device that includes a processor and a storage module. The storage module is used to store a program, and the processor is used to load and execute the program to implement an occluded pedestrian re-identification method based on pose estimation and Transformer.
[0082] An occlusion-based pedestrian re-identification method based on pose estimation and Transformer is applied to online cross-camera multi-object tracking in smart buildings to form an online cross-camera multi-object tracking method for smart buildings.
[0083] The intelligent building online multi-target tracking method across cameras includes the following steps:
[0084] A. Collect data from the building's surveillance video to obtain video clips and real-time video streams captured by multiple cameras, including one standard camera and multiple comparison cameras.
[0085] B. Use multi-target tracking methods to obtain pedestrian sequences from video clips and real-time video streams, extract pedestrian re-identification feature sequences (i.e. image sequences), and use clustering methods to obtain their representative features as comparison pedestrian images;
[0086] C. Apply the occlusion-based pedestrian re-identification method based on pose estimation and Transformer to re-identify the comparison pedestrian image with the pre-given reference pedestrian image, obtain the feature distance between the comparison pedestrian image and the reference pedestrian image. If the feature distance between the comparison pedestrian image and the reference pedestrian image is less than the set feature distance threshold, the comparison pedestrian image and the reference pedestrian image are successfully matched, and a rectangular detection box containing the entire pedestrian is obtained. Pedestrian detection information including position information and appearance information is obtained, and proceed to step E. Otherwise, the matching is marked as unsuccessful.
[0087] D. For the video stream acquired by the standard camera, the pedestrian detection information is used to continue using the multi-target tracking method to obtain the pedestrian trajectory corresponding to the baseline pedestrian image, and the feature sequence of the pedestrian trajectory is obtained to realize the smart building online cross-camera multi-target tracking method.
[0088] The multi-target tracking methods and clustering methods described in the above steps can all be implemented directly using existing technologies, so they will not be elaborated on here.
[0089] For those skilled in the art, several similar modifications and improvements can be made without departing from the inventive concept of this invention, and these should also be considered within the scope of protection of this invention.
Claims
1. An occluded person re-identification method based on pose estimation and Transformer, characterized in that, Includes the following steps: The pedestrian image is divided into regions. The pose estimation network extracts the pose estimation information of pedestrians from the pedestrian image, obtains the highest confidence score of each human keypoint region in the pedestrian image, and divides the pedestrian image into K human keypoint regions and 1 background region. The region division method includes the following steps: Set confidence threshold , Pose estimation network is used to extract pose estimation information of pedestrians. For each point in the space corresponding to the pedestrian image, the confidence level relative to K human key points is obtained. Extract the highest confidence score from the K confidence scores for each point, and determine whether the highest confidence score is greater than a threshold. If the confidence level is greater than 0, the point is assigned to the human key point region corresponding to the highest confidence level; otherwise, the point is assigned to the background region. Construct a feature extraction network based on Vision Transformer to extract the global human feature matrix of pedestrian images. Human body key point feature matrix ; Matching is performed on global human body features and keypoint human body features. The distance between global human body features and the distance between K keypoint human body features are calculated, and a weighted average is calculated between the global human body feature distance and the keypoint feature distance. The formula for calculating the distance is as follows: In the above formula Cosine distance and Given pedestrian images And the global human feature matrix of pedestrian image b, and Given pedestrian images And the feature matrix of the i-th human body key points in pedestrian image b, and Given pedestrian images The highest confidence score within the i-th keypoint region of pedestrian image b. and This is the weighting factor.
2. The occluded person re-identification method based on pose estimation and Transformer as described in claim 1, wherein, K is set to 14. The 14 key points of the human body include the left and right elbows, left and right wrists, left and right shoulders, head, neck, left and right ankles, left and right knees, and left and right hips.
3. The occluded person re-identification method based on pose estimation and Transformer according to claim 1, wherein, Extracting the global human feature matrix from pedestrian images Human body key point feature matrix The process is as follows: Divide pedestrian images into fixed-size segments. Each block is obtained through linear mapping. Each feature is divided into blocks, and the method of dividing the region is used to divide each feature into the corresponding human key point region. Create one global token and K keypoint tokens. Then, combine the global token and the K keypoint tokens... The length is obtained by concatenating the features of each block. The input sequence is fed into the encoder of the Vision Transformer; The multi-head self-attention calculation formula is constructed on the encoder of the Vision Transformer, and the formula is as follows: In the above formula, Q is the query vector, K is the key vector, V is the value vector, and D is the dimension of the query vector. It is the softmax function; The global token and all block features are mapped to the Q, K, V spaces, and multi-head attention is performed to obtain the human global feature matrix. ; The keypoint tokens and the block features within the keypoint regions corresponding to the keypoint tokens are mapped to the Q, K, and V spaces, and multi-head attention is performed to obtain the human keypoint feature matrix. .
4. The occluded person re-identification method based on pose estimation and Transformer according to claim 1 or 3, wherein, The Vision Transformer-based feature extraction network employs a triplet loss function and a classification loss function optimization model. The parameters of the Vision Transformer-based feature extraction network are adjusted based on the triplet loss function and the classification loss function. The calculation formulas for the loss functions in the triplet loss function and classification loss function optimization model include: Formula for calculating triplet loss of human keypoint features: , The formula for calculating the classification loss of human body keypoint features: , The triplet loss function and the classification loss function optimize the final loss function of the model: ， In the above formula The distance between the features of the anchor sample and the positive sample. The distance between the features of the anchor sample and the negative sample. For parameters; N is the number of sample categories. This represents the output classification probability; These are the triplet loss and classification loss for global human body features, respectively. Let the triplet loss and classification loss be the features of the i-th human body key points. and This is the weighting factor.
5. The occluded person re-identification method based on pose estimation and Transformer according to claim 4, wherein, The and The values are 0.5 and . .
6. The occluded person re-identification method based on pose estimation and Transformer according to claim 1, wherein, The and 1 and .
7. A processing apparatus comprising a processor and a storage module, the storage module being used to store a program, characterized in that, The processor is used to load and execute a program to implement the occluded pedestrian re-identification method based on pose estimation and Transformer as described in any one of claims 1 to 6.