Person Re-ID Method Based on Pose Estimation and Non-local Network

Abstract

The present invention discloses a pedestrian re-identification method based on posture estimation and non-local network. First, a pedestrian image is obtained, and a joint point heat map and a global feature map are extracted. The global feature map and the joint point heat map are element-wise multiplied to obtain local features of the joint points. The local features of the joint points are then spliced ​​and subjected to a convolution operation to obtain a first feature map. The first feature map is used to capture the correlation between the pixels of each joint point through a non-local neural network to obtain a local foreground feature map. Finally, the global feature map and the local foreground feature map are fused, and the fused feature map is subjected to global average pooling to obtain a pedestrian feature vector. The pedestrian feature vector is used for comparison and the recognition result is output. The present invention overcomes the problem of the singularity of global features and local features, and improves the accuracy of pedestrian re-identification.

Description

Technical Field

[0001] The present application belongs to the field of pedestrian re-identification technology, and in particular relates to a pedestrian re-identification method based on posture estimation and non-local network. Background Art

[0002] Person re-identification (PDR) is a subproblem of image retrieval. Given an image of a specific person captured by a single camera, the technology aims to re-identify that person from images captured by different cameras with different viewpoints. In recent years, with the continuous advancement of artificial intelligence (AI), existing video surveillance systems have also become increasingly intelligent, and intelligent systems are widely used in people's work and daily lives. Pedestrians are the main subjects in video surveillance, and their identity, location, and behavior are of particular interest in multiple research areas within intelligent surveillance. PDR is a key research area, closely related to pedestrian location and identity.

[0003] Person re-identification primarily addresses the problem of identifying and retrieving pedestrians across cameras and scenarios. Generally, given an image of a target pedestrian, person re-identification technology performs a comparative search in a library of pedestrian images captured by multiple cameras to find images with the same identity as the pedestrian. Combined with information such as the camera's location and capture time, the pedestrian's location can be quickly locked or their movement trajectory can be determined. Person re-identification distinguishes and identifies pedestrians based on their appearance features, such as their posture and jewelry. However, due to the complexity of surveillance scenarios, person re-identification faces various challenges: low camera resolution, different camera angles and parameters, different lighting conditions for pedestrians at different times, occlusion of pedestrian images due to complex scenes, and changes in the posture and clothing of different pedestrians. Therefore, extracting a more discriminative feature for person re-identification still has strong theoretical and practical value.

[0004] With the continuous maturity and development of various neural network models and computing resources, deep learning has achieved relatively good performance in image processing. Convolutional neural networks have also achieved good accuracy in various upstream and downstream computer vision tasks. Therefore, convolutional neural networks are generally used for person re-identification. Deep learning-based person re-identification mainly utilizes the global features of pedestrians. Pedestrian images are passed through a convolutional neural network to extract global pedestrian feature vectors. Each feature vector is then compared for similarity to rank and ultimately obtain the sorted results. However, due to the presence of background interference in pedestrian images, global features may focus on background and other noisy information. Therefore, it is necessary to extract local information through attention mechanisms or part-based models to reduce the interference of background changes on the results. However, both single local features and global features have certain limitations, which affect the improvement of the final accuracy of person re-identification. Summary of the Invention

[0005] The purpose of this application is to provide a pedestrian re-identification method based on posture estimation and non-local network, which overcomes the problem of the singleness of using only global features and local features, and the problem that component-based local features do not contain component relationship information.

[0006] In order to achieve the above objectives, the technical solutions of this application are as follows:

[0007] A person re-identification method based on pose estimation and non-local network, comprising:

[0008] Obtain pedestrian images, extract joint point heat maps through the posture estimation network, and extract global feature maps through the global feature extraction network;

[0009] The global feature map and the joint point heat map are element-wise multiplied to obtain the local features of the joint points. The local features of the joint points are then concatenated and subjected to a convolution operation to obtain the first feature map. The first feature map is used to capture the correlation between the pixels of each joint point through a non-local neural network to obtain a local foreground feature map.

[0010] Fuse the global feature map and the local foreground feature map to obtain a fused feature map;

[0011] The feature vector obtained by global average pooling of the fused feature map of the pedestrian image to be retrieved is compared with the feature vectors of each pedestrian image in the retrieval image library for similarity, and the recognition result is output.

[0012] Furthermore, the posture estimation network and the global feature extraction network adopt the high-resolution network HRNet-W32.

[0013] Furthermore, the extraction of joint point heat maps through the posture estimation network includes:

[0014] The output of the high-resolution network HRNet-W32 is convolved to obtain the joint point heat map.

[0015] Furthermore, extracting the global feature map through the global feature extraction network includes:

[0016] The output of the sub-network in the high-resolution network HRNet-W32 is passed through the bottleneck block with the corresponding number of input channels to generate the corresponding feature maps, and these feature maps are upsampled so that the width and height of each upsampled feature map are equal. The upsampled feature maps are then spliced ​​to obtain the final global feature map.

[0017] Furthermore, the fusion of the global feature map and the local foreground feature map to obtain a fused feature map includes:

[0018] The global feature map F gand local foreground feature map F hp Converted into three nonlinear mapping features as shown below:

[0019] F q =W q ·F hp ;

[0020] F k =P k (W k ·F g );

[0021] F v =P v (W v ·F g );

[0022] Among them, W q ,W k ,W v There are three 1×1 convolutions, P k and P v is pyramid average pooling;

[0023] F q and F k Perform dot multiplication calculation, and then use SoftMax to normalize to obtain the relationship matrix M. The formula is as follows:

[0024]

[0025] Where T is the matrix transpose;

[0026] The relationship matrix M is combined with the pyramid average pooled F v Perform matrix multiplication to obtain feature F c , F c =M·F v ;

[0027] The feature F c and the local foreground feature map F hp Splicing is performed, and then the final fusion feature map F is obtained through convolution operation h , F h =Conv(concat(F c ,F hp ));

[0028] In the formula, concat means concatenation and Conv means convolution.

[0029] This application proposes a pedestrian re-identification method based on posture estimation and non-local network, which integrates local features and global features of joint points. It can overcome the problem that local information does not contain relationship information, and overcome the singleness problem of global features and local features, thereby improving the accuracy of pedestrian re-identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of the pedestrian re-identification method based on pose estimation and non-local network in this application.

[0031] Figure 2 This is a schematic diagram of the network model structure of an embodiment of the present application.

[0032] Figure 3 This is a schematic diagram of the structure of the high-resolution network HRNet-W32 according to an embodiment of the present application.

[0033] Figure 4 This is a flowchart of extracting a local foreground feature map according to an embodiment of the present application.

[0034] Figure 5 This is a flowchart of the fusion of the global feature map and the local foreground feature map in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0036] In one embodiment, Figure 1 、 Figure 2 As shown in the figure, a pedestrian re-identification method based on pose estimation and non-local network is proposed, including:

[0037] Step S1: Obtain a pedestrian image, extract joint point heat maps through a posture estimation network, and extract a global feature map through a global feature extraction network.

[0038] This application uses a trained global feature extraction network and posture estimation network to extract global feature maps and joint point heat maps.

[0039] During training, after acquiring pedestrian images, data preprocessing can be performed on them through horizontal flipping and erasing to enrich the training samples. During data preprocessing, each pedestrian image is first resized to a height of 256 and a width of 128. A threshold of 0.5 is given, and a horizontal flip probability is generated for each pedestrian image. When the probability exceeds the threshold, the left and right halves of the image are horizontally flipped about the vertical center axis of the pedestrian image. A random erasure probability is also generated for each pedestrian image. When the probability exceeds the threshold, a rectangular region with a height of 118 and a width of 88 is randomly selected from the pedestrian image, and the pixel values ​​of each channel in this region are set to 0.

[0040] After training the high-resolution network, for pedestrian images to be identified, you only need to resize the pedestrian images to images with a height of 256 and a width of 128, and input them into the trained high-resolution network to extract features.

[0041] After acquiring a pedestrian image, it is fed into a global feature extraction network and a posture estimation network, respectively, to extract a global feature map and a joint point heat map. In this embodiment, both the global feature extraction network and the posture estimation network use the high-resolution network HRNet-W32.

[0042] In a specific embodiment, extracting a joint point heat map through a posture estimation network includes:

[0043] The output of the high-resolution network HRNet-W32 is convolved to obtain the joint point heat map.

[0044] The global feature map is extracted through the global feature extraction network, including:

[0045] The output of the sub-network in the high-resolution network HRNet-W32 is passed through the bottleneck block with the corresponding number of input channels to generate the corresponding feature maps, and these feature maps are upsampled so that the width and height of each upsampled feature map are equal. The upsampled feature maps are then spliced ​​to obtain the final global feature map.

[0046] like Figure 3 As shown in Figure 2. For the posture estimation network, the output of the high-resolution network HRNet-W32 is convolved with 1×1 to obtain a joint point heat map with 13 channels. For the global feature extraction network, the outputs of the four sub-networks in the high-resolution network HRNet-W32 {X 32 ,X 64 ,X 128 ,X 256 Generate the corresponding feature map {X through the bottleneck block corresponding to the number of input channels 128 ,X256 ,X 512 ,X 1024}, and upsample these feature maps so that the width W and height H corresponding to each feature map are equal, and then these feature maps are spliced ​​to obtain the final global feature map F with 1920 channels g .

[0047] Step S2: Multiply the global feature map and the joint point heat map element-by-element to obtain the local features of the joint points. Then, concatenate the local features of the joint points and perform a convolution operation to obtain the first feature map. The first feature map is used to capture the correlation between the pixels of each joint point through a non-local neural network to obtain a local foreground feature map.

[0048] Specifically, such as Figure 2 As shown in the figure, firstly, the global feature map is multiplied with the joint point heat map to obtain the local features of the joint points.

[0049] In this embodiment, the global feature map is convolved with 1×1 and the dimension is reduced to obtain a feature map with 256 channels. The feature map is then multiplied with the joint point heat map to obtain 13 joint point local features {F p1 ,F p2 ,…,F p13}.

[0050] Then, the local foreground feature map F is obtained through the FNL (Foreground Non local module) hp , specifically including:

[0051] Step F1: Concatenate the local features of each joint point and perform a convolution operation to obtain a first feature map.

[0052] The local features of each joint are spliced ​​together, and a 1×1 convolution W1 is used to generate the first feature map F containing each joint point. p , the formula is as follows:

[0053] F p =W1(concat(F p1 ,F p2 ,…,F p13 ))

[0054] Among them, concat represents concatenation and W1 represents 1×1 convolution.

[0055] Step F2: Use the non-local neural network to capture the correlation between the pixels of each joint point in the first feature map to obtain local foreground features.

[0056] This step uses standard non-local neural networks to capture the correlation between pixels at each joint point, expand the receptive field, and obtain discriminative human foreground semantic information.

[0057] Specifically, such as Figure 4 As shown, F p Input two 1×1 convolutional layers respectively to obtain two new feature maps, and then retransform them into matrices Q and K of size N×C and C×N, where the size of N is as follows:

[0058] N=H×W

[0059] N represents the total number of pixels in each channel, H and W represent the height and width of the feature map, and C represents the number of channels.

[0060] Use matrix multiplication of matrices Q and K to calculate the relationship mapping between them, and use the SoftMax operation to normalize the relationship mapping, as shown in the following formula:

[0061] M′=SoftMax(Q·K)

[0062] A point (i, j) in M′ represents the correlation between the i-th pixel and the j-th pixel.

[0063] In addition, F p The input is fed into another 1×1 convolution to generate a feature map and re-transformed into a matrix V of size N×C, which is then multiplied by M to integrate the relational mapping into the original V to obtain a feature map and combine it with F. p After the residual connection, the final local foreground feature map F is obtained through 1×1 convolution. hp , the formula is as follows.

[0064] F hp =Conv(V·M+F p )

[0065] Where Conv represents convolution and + represents residual connection. In this way, the local foreground feature map can effectively aggregate the information of all joint points.

[0066] Step S3: Fuse the global feature map and the local foreground feature map to obtain a fused feature map.

[0067] This step fuses the global feature map F h and local foreground feature map F hp Get the fusion feature map F h ,Right now Figure 2The operations performed by HPGEF (Hybrid Pose and global feature fusion module) in .

[0068] Specific as Figure 5 Shown, including:

[0069] Step S3.1: transform the global feature map F g and local foreground feature map F hp Converted into three nonlinear mapping features as shown below:

[0070] F q =W q ·F hq ;

[0071] F k =P k (W k ·F g );

[0072] F v =P v (W v ·F g );

[0073] Among them, W q ,W k ,W v There are three 1×1 convolutions, P k and P v It is pyramid average pooling, which uses this pooling method to reduce the computational overhead of the non-local module itself, embed contextual information at different scales into the network, and obtain multi-scale pooling results.

[0074] The obtained F q ∈C′×(H×W),F k ∈C′×S,F v ∈C′×S, S represents the pixels of pyramid average pooling.

[0075] Step S3.2, F q and F k Perform dot product calculation and then use SoftMax to normalize to obtain the relationship matrix M.

[0076] This step uses matrix operations as the distance metric. q and F k The pixel relationship in is calculated, and then normalized with SoftMax to obtain the relationship matrix M. The pixel points in the relationship matrix M are converted to between 0 and 1. The formula is as follows:

[0077]

[0078] Where T is the matrix transpose.

[0079] Step S3.3, the relationship matrix M is combined with the pyramid average pooled F v Perform matrix multiplication to obtain feature F c .

[0080] The obtained M∈S×(H×W) is then combined with the pyramid average pooled F v Perform matrix multiplication to obtain F c , the formula is as follows:

[0081] F c =M·F v

[0082] F c Each pixel in F g The corresponding F hp The weights are obtained from F g are selected from all pixels.

[0083] Step S3.4: Set the feature F c and the local foreground feature map F hp Splicing is performed, and then the final fusion feature map F is obtained through convolution operation h .

[0084] The feature F c and the local foreground feature map F hp After splicing, the final output result F is obtained by 1×1 convolution. h , the formula is as follows:

[0085] F h =Conv(concat(F c ,F hp ))

[0086] In the formula, concat means concatenation and Conv means convolution. h As the final output feature map, and perform global average pooling ( Figure 2 After GAP), the feature vector is obtained as the input of the similarity measurement.

[0087] Step S4: perform global average pooling on the fusion feature map of the pedestrian image to be retrieved to obtain a feature vector, perform similarity comparison with the feature vectors of each pedestrian image in the retrieval image library, and output the recognition result.

[0088] For the pedestrian image to be identified, it is resized and input into the network of this application, and the fused feature map is output. Then, global average pooling is performed to obtain the feature vector. Each image in the retrieval image library is also pre-processed through the network of this application to obtain the corresponding feature vector and saved in the database. The feature vector obtained from the pedestrian image to be identified is compared with the feature vectors of each pedestrian image in the retrieval image library for similarity, and the final ranking result is output based on the feature similarity. The most similar image is selected as the final recognition result.

[0089] In this application, the technical solution of this application is experimentally verified. In the experiment, based on the Market1501 dataset and the DukeMTMC-reID dataset, the current mainstream pedestrian re-identification algorithm is compared, and the mean average precision (mAP) and the k-th hit rate (Rank-k) are used to measure the recognition performance. The experimental results are shown in Tables 1 and 2:

[0090]

[0091] Table 1

[0092]

[0093] Table 2

[0094] Table 1 and Table 2 are the test results on the Market1501 dataset and the DukeMTMC-reID dataset, respectively. By comparing the two tables, it can be seen that the method of this application is superior to other pedestrian re-identification algorithms in 2019, 2020 and 2021 in pedestrian re-identification. The mAP and Rank-k indicators are both optimal. Therefore, the method of this application has certain advantages over other algorithms.

[0095] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A person re-identification method based on pose estimation and non-local network, characterized in that: The pedestrian re-identification method based on posture estimation and non-local network includes: Obtain pedestrian images, extract joint point heat maps through the posture estimation network, and extract global feature maps through the global feature extraction network; The global feature map and the joint point heat map are element-wise multiplied to obtain the local features of the joint points. The local features of the joint points are then concatenated and subjected to a convolution operation to obtain the first feature map. The first feature map is used to capture the correlation between the pixels of each joint point through a non-local neural network to obtain a local foreground feature map. Fuse the global feature map and the local foreground feature map to obtain a fused feature map; The feature vector obtained by global average pooling of the fused feature map of the pedestrian image to be retrieved is compared with the feature vectors of each pedestrian image in the retrieval image library for similarity, and the recognition result is output; The posture estimation network and global feature extraction network adopt the high-resolution network HRNet-W32; The extracting of the global feature map by the global feature extraction network includes: The output of the sub-network of the high-resolution network HRNet-W32 is passed through the bottleneck block corresponding to the number of input channels to generate the corresponding feature maps. These feature maps are then upsampled so that the width and height of each upsampled feature map are equal. The upsampled feature maps are then concatenated to obtain the final global feature map. The fusing of the global feature map and the local foreground feature map to obtain a fused feature map includes: The global feature map F g and local foreground feature map F hp Converted into three nonlinear mapping features as shown below: F q =W q ·F hp ; F k =P k (W k ·F g ); F v =P v (W v ·F g ); Among them, W q , W k , W v There are three 1×1 convolutions, P k and P v is pyramid average pooling; F q and F k Perform dot multiplication calculation, and then use SoftMax to normalize to obtain the relationship matrix M. The formula is as follows: Where T is the matrix transpose; The relationship matrix M is combined with the pyramid average pooled F v Perform matrix multiplication to obtain feature F c , F c =M·F v ; The feature F c and the local foreground feature map F hp Splicing is performed, and then the final fusion feature map F is obtained through convolution operation h , F h =Conv(concat(F c , F hp )); In the formula, concat means concatenation and Conv means convolution.

2. The person re-identification method based on pose estimation and non-local network according to claim 1, characterized in that: The extraction of joint point heat maps through the posture estimation network includes: The output of the high-resolution network HRNet-W32 is convolved to obtain the joint point heat map.

Images