An occluded pedestrian re-identification method based on pose guidance and dynamic feature extraction

Abstract

The application discloses a kind of posture guidance and dynamic feature extraction's occluded pedestrian re-identification method;The main difficulty of occluded pedestrian re-identification is due to the phenomenon of occlusion, the body of pedestrian is often incomplete, and the pedestrian features obtained from the image contain background noise.The application effectively utilizes the human pose estimation network to extract human key points, and automatically extracts the unoccluded part of the human body from the image block output by the encoder using the matching method, optimizes the alignment operation of the unoccluded part, and reduces the interference of the occlusion to the image.In addition, the application effectively utilizes the characteristics of attention mechanism and convolution operation, generates a dynamic unoccluded mask for the image with the help of posture information, realizes dynamic alignment of the unoccluded part, and maintains intra-class compactness and inter-class dispersion through angola barrier loss.The application combines metric loss and classification loss for end-to-end network training, which fully improves the training efficiency and training stability.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, specifically relating to a pedestrian re-identification method in occluded scenarios based on pose guidance and dynamic feature extraction. Background Technology

[0002] Pedestrian re-identification is a subtask of image retrieval in computer vision. Given an image of a pedestrian, pedestrian re-identification methods can find images of the same pedestrian at other times and locations. With the widespread use of surveillance cameras, pedestrian re-identification is of great significance for realizing smart security and urban brains. Existing pedestrian re-identification mainly includes two steps: 1) extracting image features to characterize pedestrians; 2) measuring image features, such as common Euclidean distance and cosine distance. With the development of deep learning, researchers have proposed deep learning-based pedestrian re-identification methods that can be trained end-to-end to learn image features that are robust and discriminative to pedestrian appearances. However, existing methods perform poorly in pedestrian re-identification under occluded conditions.

[0003] The main challenge in identifying occluded pedestrians lies in the fact that, due to occlusion, the pedestrian's body is often incomplete, and the pedestrian features obtained from the image are mixed with background noise. Comparing these features with other pedestrian features presents difficulties, leading to unsatisfactory results. Summary of the Invention

[0004] The purpose of this invention is to provide a method for re-identifying occluded pedestrians based on posture guidance and dynamic feature extraction.

[0005] In a first aspect, the present invention provides a method for pedestrian re-identification in occluded scenarios using posture guidance and dynamic feature extraction, comprising the following steps:

[0006] Step 1: Preprocess the image to be identified to obtain a two-dimensional matrix.

[0007] Step 2: Input the two-dimensional matrix obtained in Step 1 into the encoder; extract the global and local features of the image to be recognized.

[0008] Step 3: Extract human pose key point information through human pose estimation network to obtain the location and confidence of human key points, as well as the heat map generated from the location.

[0009] Step 4: Obtain local features for posture guidance based on local features and key points of human posture.

[0010] 4-1. Merge the last two dimensions of the heatmap obtained in step 3 and pass them through a fully connected layer.

[0011] 4-2. Multiply the heatmap and local features element by element to obtain a set of pose guidance features.

[0012] 4-3. Calculate the cosine distance between each attitude guidance feature and each set of local features, and match the nearest set of local features for each heatmap.

[0013] 4-4. Add the corresponding local features to each posture guidance feature to obtain the local features of posture guidance.

[0014] Step 5: Perform global average pooling on the heatmap obtained in Step 3 and multiply it with the global features extracted in Step 2 to obtain image features with embedded pose information.

[0015] Step 6: Extract the image dynamic feature mask of the image to be identified using a dynamic feature generator.

[0016] Step 7: After removing the parts with confidence scores below the threshold from the local features of the pose guidance obtained in Step 4, concatenate them with the image features with embedded pose information obtained in Step 5 and the image dynamic feature mask obtained in Step 6 to obtain the final image feature. Calculate the distance between this feature and all image features in the image library and sort them in ascending order. Take the identity information of the image feature with the smallest distance as the identity of the pedestrian in the identified image.

[0017] As a preferred option, the process of step 1 is as follows:

[0018] 1-1. Preprocess the image to be identified and adjust it to a standard image to be identified.

[0019] 1-2. The standard image to be identified is transformed by a convolutional layer, and the first two dimensions of the resulting features are merged to obtain a two-dimensional matrix.

[0020] 1-3. Add the same length of tile position markers and camera position markers to the two-dimensional matrix obtained in step 1-2, and then concatenate it with a learnable classification marker.

[0021] Preferably, in step 2, the encoder uses a Vision Transformer network, which includes 12 self-attention layers. Before inputting into the last self-attention layer, the features other than the classification label are divided into multiple groups. Each group of features is input into a weight-shared self-attention layer, and after passing through a batch normalization layer, multiple groups of local features are obtained, with the classification label serving as the global feature.

[0022] Preferably, in step 3, the human pose estimation network performs a convolution operation on the image to be identified, encoding the location information of human key points into multiple sets of heatmaps, representing multiple human pose key point information of the human body.

[0023] Preferably, the human pose estimation network uses the OpenPose network.

[0024] As a preferred option, the specific process of step 6 is as follows:

[0025] 6-1. Concatenate the remaining features from the output features of layers 1, 3, 5, 7, 9, and 11 of the encoder, excluding the classification label.

[0026] 6-2. Input the feature map obtained in step 6-1 into a dynamic feature generator consisting of 6 sets of convolutional layers. Each set of convolutional layers reduces the number of channels by half, then performs max pooling, and finally passes through a fully connected layer to obtain a dynamic human feature mask.

[0027] Preferably, the expression for the total loss L during model training of this identification method is as follows:

[0028] L=λ1L cls +λ2L tri +L A-cls

[0029] Among them, L cls The cross-entropy loss is expressed as follows:

[0030]

[0031] B is the number of images in a batch during training; N is the total number of pedestrians during training; P j y is the probability that the network predicts the identified image to be of class j, and y is the actual class of the identified image.

[0032] L tri The triplet loss is expressed as follows:

[0033] L tri =max(D(f) a f p )-D(f a f n )+m,0)

[0034] f a f p f n These are the anchor image, the positive sample image of the same category as the anchor image, and the negative sample image of a different category from the anchor image; D(·) is the Euclidean distance calculation function; m is the barrier coefficient.

[0035] L A-cls The blocking loss for Angola is expressed as follows:

[0036]

[0037]

[0038] N is the total number of pedestrian IDs during training; s is the hyperparameter for adjusting the loss scale; m is the blocking coefficient; F Pose It is a global image feature of pose embedding; D yi It is the dynamic mask of the image with pedestrian label yi, and cos(·) is the cosine distance calculation function.

[0039] In a second aspect, the present invention provides a computer device comprising a memory and at least one processor; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the aforementioned identification method.

[0040] Thirdly, the present invention provides a readable storage medium storing computer instructions; when executed by a processor, the computer instructions are used to implement the identification method as described above.

[0041] The beneficial effects of this invention are:

[0042] 1. This invention effectively utilizes a human pose estimation network to extract human key points and uses a matching method to automatically extract the unoccluded parts of the human body from the image blocks output by the encoder, optimizing the alignment operation of the unoccluded parts and reducing the interference of occlusions on the image.

[0043] 2. This invention effectively utilizes the characteristics of attention mechanisms and convolution operations, generating dynamic unoccluded masks for images with the help of pose information, achieving dynamic alignment of unoccluded parts, and maintaining intra-class compactness and inter-class dispersion through Angora blocking loss. This invention combines metric loss and classification loss for end-to-end network training, significantly improving training efficiency and stability. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the process of the present invention.

[0045] Figure 2 This is a schematic diagram of the model structure of the present invention. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings.

[0047] like Figure 1 and 2 As shown, a pedestrian re-identification method for occluded scenarios with posture guidance and dynamic feature extraction is presented. The method uses a pedestrian re-identification model for identification. The pedestrian re-identification model includes an encoder, a human posture estimation network, a local feature aggregation module, a global feature aggregation module, and a dynamic feature generator.

[0048] The pedestrian re-identification method in this occluded scenario includes the following steps:

[0049] Step 1: Image preprocessing.

[0050] 1-1. Preprocess the image to be identified and adjust it to a standard image size of 256×128×3.

[0051] 1-2. The dimensions of the standard image being recognized are converted to 14×14×768 by a convolutional layer with a kernel size of 16×16, a stride of 16, and 768 channels. Then the first two dimensions are merged to form a two-dimensional matrix of 196×768.

[0052] 1-3. Add the same length tile location token and camera location token to the two-dimensional matrix obtained in step 1-2, and then concatenate it with a learnable classification token. The final two-dimensional matrix input into the encoder has a dimension of 197×768.

[0053] Step 2: Extract global and local features using the encoder.

[0054] The encoder consists of 12 self-attention layers. Before being input to the last self-attention layer, the features, excluding the classification token, are divided into N groups, where N ranges from 5 to 20. Each group of features is input into a weight-shared self-attention layer, and after passing through a batch normalization layer, N groups of local features are obtained, with the classification token serving as the global feature. The size of the local features is K×768.

[0055] Optionally, the steps of the encoder to extract global features of the image and local features of image groups are as follows:

[0056] (1) The pedestrian image is cropped into N non-overlapping patches through a sliding window of fixed size, and then the input sequence is obtained through a fully connected layer.

[0057] (2) Add a location embedding token and a camera location embedding token to the input sequence, and then concatenate them with a learnable classification token to form the encoder input matrix. The input matrix includes a classification token and a two-dimensional matrix with added location embedding information.

[0058] (3) After the encoder extracts the features, the features can be divided into global image features Fg extracted from the classification token and local features Fp extracted from the rest of the image.

[0059] (4) In the last layer of the encoder, Fp is divided into K groups, and each group is passed through a weight-shared self-attention layer to obtain K image group local features.

[0060] Step 3: Extract human pose key point information through human pose estimation network.

[0061] An existing human pose estimation network is used to perform convolution operations on the standard image to be identified. This allows the network to encode the location information of human key points into M heatmaps, representing the M key point information of the human body. The human estimation network uses the OpenPose network. The heatmap size is M×64×32; to integrate local features and heatmaps, K=M.

[0062] Step 4: The local feature aggregation module obtains a set of posture-guided local features based on local features and human posture key point information.

[0063] 4-1. Merge the last two dimensions of the heatmap, at which point the heatmap size is M×2048. Then, use a fully connected layer to change the heatmap size to M×768.

[0064] 4-2. Multiply the heatmap and local features element-wise to obtain a set of pose guidance features with a size of M×768.

[0065] 4-3. Calculate the cosine distance between each pose guidance feature and each set of local features, and match the closest set of local features for each heatmap. This set of local features represents the features most likely to represent a certain human keypoint in the entire image. Based on this, the human features of that keypoint can be extracted.

[0066] 4-4. Add the corresponding local features to each pose guidance feature to obtain a set of aggregated pose guidance local features.

[0067] Step 5: The global feature aggregation module obtains image features with embedded pose information based on human pose key point information and global features.

[0068] After performing global average pooling on the M heatmaps obtained in step 3, multiply them with the global features extracted by the encoder in step 2 to obtain image features with embedded pose information.

[0069] Step 6: Extract the dynamic feature mask of the image using a dynamic feature generator.

[0070] 6-1. Concatenate the output features of layers 1, 3, 5, 7, 9, and 11 of the encoder, excluding the classification token, to obtain a feature map of size 6×196×768. Adjust the size of this feature map to 768*6×w×h.

[0071] 6-2. Input the obtained feature map into a dynamic feature generator consisting of 6 sets of convolutional layers. Each set of convolutional layers reduces the number of channels by half, then performs max pooling, and finally passes through a fully connected layer to obtain a dynamic human feature mask of size 1×768.

[0072] Step 7: After removing the parts with confidence below the threshold from the pose-guided local features obtained in Step 4, concatenate them with the image features with embedded pose information obtained in Step 5 and the image dynamic feature mask obtained in Step 6 to obtain the final image features. Calculate the distance between this feature and all image features in the image library and sort them in ascending order. Use the sorting result as the output of the pedestrian re-identification model.

[0073] The pose estimation network generates human keypoint information by providing the location and confidence score for each keypoint. The confidence score ranges from 0 to 1. When the confidence score is greater than a threshold γ, the keypoint is considered visible. The pose-guided local features with confidence scores greater than the threshold γ are used as the final feature generation. Here, the threshold γ = 0.5.

[0074] Distance metrics between pedestrian image features can be Euclidean distance, cosine distance, or any other reasonable distance metric. The features of each image in the image database are calculated offline and stored after the model is trained. Therefore, when querying pedestrian images, only the query image needs to be fed into the network to extract features, and then its distance to all images in the database can be quickly calculated.

[0075] The training process of the pedestrian re-identification model is as follows:

[0076] (1) Obtain the pedestrian re-identification dataset.

[0077] The dataset is divided into a training set, a query set, and a query set. During the training phase, each batch contains k images of P randomly selected pedestrians as the images to be identified.

[0078] (2) Extract global and local features of the image through the encoder.

[0079] (3) Extract human pose key point information through pose estimation network.

[0080] (4) The local features and human pose key point information are combined through a feature aggregator to obtain pose-guided human key point features.

[0081] (5) The global features and human pose key point information are used to obtain image features with embedded pose information through the feature embedding layer.

[0082] (6) Extract the dynamic feature mask of the image through the dynamic feature generator.

[0083] (7) After performing global average pooling on the human keypoint features guided by posture, calculate the classification loss and triplet loss.

[0084] (8) Calculate the cross-entropy loss and triplet loss for the image features with embedded pose information.

[0085] The cross-entropy loss is:

[0086]

[0087] Where B is the number of images in a batch during training; N is the total number of pedestrian IDs during training; P j y is the probability that the network predicts the identified image to be of class j, and y is the actual class of the identified image.

[0088] The triplet loss is:

[0089] L tri =max(D(f) a f p )-D(f a f n )+m,0)

[0090] Among them, f a f p f n These are the anchor image, the positive sample image of the same category as the anchor image, and the negative sample image of a different category from the anchor image; D(·) is the Euclidean distance calculation function; m is the barrier coefficient.

[0091] (9) Calculate the Angolan blocking loss using image features and dynamic feature masks with embedded attitude information.

[0092] The Angolan barrier loss is:

[0093]

[0094]

[0095] Where B is the number of images in a batch during training; N is the total number of pedestrian IDs during training; s is the hyperparameter for adjusting the loss scale; m is the blocking coefficient; F Pose It is a global image feature of pose embedding; D yi It is the dynamic mask of the image with pedestrian label yi, and cos(·) is the cosine distance calculation function.

[0096] (10) The total loss L during the training phase is the cross-entropy loss L. cls Triple loss L tri Angola barrier loss L A-cls The weighted sum, specifically expressed as follows:

[0097] L=λ1L cls +λ2L tri +L A-cls

[0098] Here, λ1 and λ2 are hyperparameters for adjusting the loss scale.

[0099] (11) Train the pedestrian re-identification model with the goal of minimizing the total loss L.

[0100] This invention provides a storage device storing multiple programs, which are adapted to be loaded by a processor and implement the above-described method for pedestrian re-identification in occluded scenarios with posture guidance and dynamic feature extraction.

[0101] This invention provides a processing device, including a processor and a storage device; the processor is adapted to execute various programs; the storage device is adapted to store multiple programs; the programs are adapted to be loaded and executed by the processor to implement the above-described method for pedestrian re-identification in occluded scenarios with posture guidance and dynamic feature extraction.

Claims

1. A method for re-identifying occluded pedestrians based on posture guidance and dynamic feature extraction, characterized in that: Includes the following steps: Step 1: Preprocess the image to be identified to obtain a two-dimensional matrix; Step 2: Input the two-dimensional matrix obtained in Step 1 into the encoder; extract the global and local features of the image to be recognized; The encoder uses a Vision Transformer network, which includes 12 self-attention layers. Before being input into the last self-attention layer, the features, excluding the classification label, are divided into multiple groups. Each group of features is input into a weight-shared self-attention layer, and after passing through a batch normalization layer, multiple groups of local features are obtained, with the classification label serving as the global feature. Step 3: Extract human pose key point information through human pose estimation network to obtain the location and confidence of human key points, as well as the heat map generated from the location. The human pose estimation network performs convolution operations on the image being identified, encoding the location information of human key points into multiple sets of heatmaps, representing multiple human pose key point information of the human body. Step 4: Obtain local features for posture guidance based on local features and key points of human posture; 4-1. Merge the last two dimensions of the heatmap obtained in step 3 and pass them through a fully connected layer; 4-2. Multiply the heatmap and local features element-wise to obtain a set of pose guidance features; 4-3. Calculate the cosine distance between each pose guidance feature and each set of local features, and match the nearest set of local features for each heatmap; 4-4. Add the corresponding local features to each posture guidance feature to obtain the local features of posture guidance; Step 5: Perform global average pooling on the heatmap obtained in Step 3 and multiply it with the global features extracted in Step 2 to obtain image features with embedded pose information. Step 6: Extract the image dynamic feature mask of the image to be identified using a dynamic feature generator; the specific process of step 6 is as follows: 6-1. Concatenate the remaining features from the output features of layers 1, 3, 5, 7, 9, and 11 of the encoder, excluding the classification label. 6-2. Input the feature map obtained in step 6-1 into a dynamic feature generator consisting of 6 sets of convolutional layers. Each set of convolutional layers reduces the number of channels by half, then performs a max pooling layer, and finally passes through a fully connected layer to obtain a dynamic human feature mask. Step 7: After removing the parts with confidence scores below the threshold from the local features of the pose guidance obtained in Step 4, concatenate them with the image features with embedded pose information obtained in Step 5 and the image dynamic feature mask obtained in Step 6 to obtain the final image feature. Calculate the distance between this feature and all image features in the image library and sort them in ascending order. Take the identity information of the image feature with the smallest distance as the identity of the pedestrian in the identified image.

2. The occluded pedestrian re-identification method based on posture guidance and dynamic feature extraction according to claim 1, characterized in that: The process of step 1 is as follows: 1-1. Preprocess the image to be identified and adjust it to a standard image to be identified; 1-2. The standard image to be identified is transformed by a convolutional layer, and then the first two dimensions of the resulting features are merged to obtain a two-dimensional matrix; 1-3. Add the same length tile position markers and camera position markers to the two-dimensional matrix obtained in step 1-2, and then concatenate it with a learnable classification marker.

3. The occluded pedestrian re-identification method based on posture guidance and dynamic feature extraction according to claim 1, characterized in that: The human pose estimation network described above uses the OpenPose network.

4. The occluded pedestrian re-identification method based on posture guidance and dynamic feature extraction according to claim 1, characterized in that: The total loss of this identification method during model training. L The expression is as follows: ； in, L cls The cross-entropy loss is expressed as follows: ； B It refers to the number of images in a batch during training; N It refers to the total number of pedestrians during training; P j The network predicts that the image to be identified is the first one. j The probability of a class y It is the actual category of the image being identified; L tri The triplet loss is expressed as follows: ； f a , f p , f n These are the anchor image, the positive sample image of the same category as the anchor image, and the negative sample image of a different category from the anchor image; D(·) is the Euclidean distance calculation function; m is the barrier coefficient; L A-cls The blocking loss for Angola is expressed as follows: ； N This is the total number of pedestrian IDs during training; s It is a hyperparameter for adjusting the loss scale; m is the barrier coefficient; F Pose It is a global image feature embedded in the pose; D yi The pedestrian label is yi The dynamic mask of the image, cos(·) is the cosine distance calculation function.

5. A computer device, comprising a memory and at least one processor; characterized in that: The memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, causing the at least one processor to perform the identification method as described in any one of claims 1-4.

6. A readable storage medium storing computer instructions; characterized in that: When the computer instructions are executed by the processor, they are used to implement the identification method as described in any one of claims 1-4.

Images