A joint optimization method for person re-identification integrating appearance information

Abstract

The present invention provides a joint optimization method for pedestrian re-identification that integrates appearance information, including the following steps: pre-processing a cross-appearance pedestrian re-identification dataset using a data enhancement strategy; generating batch images of multiple identities and multiple appearances according to the identity attributes and appearance attributes of the pedestrians in the image; extracting a high-dimensional feature matrix using a feature extraction network, pooling to obtain pedestrian high-dimensional features, and batch normalization to obtain pedestrian reasoning features; connecting different fully connected layers based on pedestrian reasoning features to obtain identity classification features and appearance classification features respectively; calculating the most difficult ternary loss, multi-appearance ternary loss, identity classification loss, and appearance classification loss, weighted summing the four losses and jointly optimizing them end-to-end; back-propagating the gradient of the loss layer to update the weight parameters; iterative calculation until the feature extraction network model converges. The present invention improves the comprehensive retrieval performance of the model in the same appearance and cross-appearance environments.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning, and in particular relates to a joint optimization method for pedestrian re-identification integrating appearance information. Background Art

[0002] In recent years, with the rapid development of my country's economy and science and technology, the construction of smart cities has become an important foundation for maintaining social stability and people's livelihood. Among them, pedestrian re-identification, as an important part of intelligent security, is closely related to the work, study and life of the people, and has extremely high application value in the intelligent society with an increasingly large monitoring network. Pedestrian re-identification can associate the same target pedestrians in different times, different places, and different monitoring devices. In real life, staff often need to rely on human eyes to find relevant clues in massive video data, which consumes a lot of manpower and material resources. With the advancement of deep learning algorithms, pedestrian re-identification technology based on deep networks has gradually attracted the attention of researchers, and the indicators on relevant public data sets have been continuously surpassed. However, most of these pedestrian re-identification research works are based on the assumption that pedestrians will not change their clothes. The cross-appearance pedestrian re-identification research based on appearance changes such as clothing, hair, and decorations is still in its early stages. In fact, this cross-appearance pedestrian re-identification has broad application prospects in the current big data era of the Internet of Everything, such as the search for suspects with anti-tracking awareness in the field of criminal investigation, unmanned supermarkets, album clustering, etc.

[0003] The existing long-term cross-appearance benchmark datasets under surveillance conditions include NKUP+ and PRCC, which contain 40,217 and 33,698 pedestrian images respectively, and their data are all derived from school surveillance images. As for the cross-appearance pedestrian re-identification algorithm, the current research is mainly based on identity labels in improving the feature extraction network structure (such as CCAN, 3APF), introducing additional contour and component semantic prior knowledge (such as SPT, FSAM), and generating more pedestrian data with clothing changes (such as CASENet, DGNet). The existing pedestrian re-identification model is only trained based on identity supervision information and lacks prior knowledge of appearance attributes. Summary of the invention

[0004] In order to solve the technical problems existing in the prior art, the present invention provides a joint optimization method for person re-identification integrating appearance information. The method maintains the stability of cross-appearance model training and the robustness of pedestrian retrieval features based on a batch sampling algorithm and loss function of appearance attributes. At the same time, a batch normalization structure is adopted based on multiple loss functions to jointly optimize the network model, thereby improving the comprehensive retrieval performance of the model in the same-appearance and cross-appearance environments.

[0005] The technical solution adopted by the present invention is: a joint optimization method for pedestrian re-identification integrating appearance information, comprising the following steps:

[0006] Step 1: Preprocess the cross-appearance person re-identification dataset using data augmentation strategies;

[0007] The images in the training set are preprocessed by scaling, random horizontal flipping, padding, random cropping, mean and variance subtraction, and random erasing. The images in the test set are preprocessed by scaling and mean and variance subtraction.

[0008] Step 2: Generate batch images with multiple identities and appearances based on the identity attributes and appearance attributes of the pedestrians in the preprocessed images;

[0009] Each batch of images contains a fixed number of pedestrians, each pedestrian contains a fixed number of appearances, and each appearance contains a fixed number of images. Sampling images based on the identity attributes and appearance attributes of pedestrians ensures that each pedestrian in each batch during network training will be composed of a fixed number of randomly selected images of different appearance categories, so that the network training can take into account the diversity of pedestrians' appearance, posture, lighting, background, etc., thereby improving the stability of model training and the robustness of features.

[0010] Step 3: Use the feature extraction network to extract the high-dimensional feature matrix and perform pooling to obtain the high-dimensional features of pedestrians;

[0011] Step 4: Perform batch normalization based on pedestrian high-dimensional features to obtain pedestrian inference features;

[0012] Step 5: Connect different fully connected layers based on pedestrian inference features to obtain identity classification features and appearance classification features respectively; respectively enable the network to learn to distinguish pedestrian identity information and fine-grained appearance information;

[0013] The high-dimensional features of pedestrians are first batch normalized and then input into the fully connected layer to obtain identity and appearance classification features. The reason is that the most difficult ternary loss and multi-appearance loss optimize features in the Euclidean distance space, while the identity and appearance classification losses use cosine distance, which is more reasonable. Therefore, by adding batch normalization, the distribution of features in the embedding space can be smoothed and the intra-class compactness can be increased, thereby combining the advantages of two different optimization objectives.

[0014] Step 6: Calculate the most difficult triplet loss and multi-appearance triplet loss based on the pedestrian high-dimensional features, and calculate the identity classification loss and appearance classification loss based on the identity classification features and appearance classification features respectively, then weighted sum the four losses and jointly optimize them end-to-end;

[0015] The most difficult three-dimensional loss L HardTri :

[0016]

[0017] Among them, α represents the interval parameter, D represents the distance metric, represents the kth image of the pth person in the batch High-dimensional features, 1≤p≤P, 1≤k≤K; k′ represents the k′th image, p′ represents the p′th person;

[0018] Multi-appearance ternary loss L MATri :

[0019]

[0020] in, The kth image representing the ath appearance of the pth person in the batch , 1≤p≤P, 1≤a≤A, 1≤k≤K, and pb represents the bth appearance of the pth person.

[0021] Identity classification loss L ID :

[0022]

[0023] Appearance classification loss L AP :

[0024]

[0025] Among them, x i (1≤i≤N), y i 、z i denotes the image and its identity category and appearance category respectively, p(y i |x i ) and p(z i |x i ) represent the image x i Recognized by the model as identity category y i and appearance category z i probability.

[0026] Overall loss L:

[0027] L=L HardTri +L ID +λ1L MATri +λ2L AP

[0028] Among them, λ1 and λ2 represent the weight parameters of multi-appearance triplet loss and appearance classification loss, respectively.

[0029] Both λ1 and λ2 are set to 0.2.

[0030] In addition to the identity classification loss and the most difficult triplet loss commonly used in person re-identification, the overall loss also designs multi-appearance triplet loss and appearance classification loss based on fixed-structure batch images and pedestrian appearance attributes. Multi-appearance triplet loss enables the network to learn to close the distance between different appearance image features of the same person, while appearance classification loss directly learns the appearance information of pedestrians. The combination of the two promotes the model to learn robust pedestrian retrieval capabilities in the same appearance and cross-appearance comprehensive environments.

[0031] Step 7: Back propagate the gradient of the loss layer to update the weight parameters of the feature extraction network and the fully connected layer;

[0032] Step 8: Repeat steps 2-7 until the feature extraction network model converges or the maximum number of iterations is reached to obtain a multi-loss joint optimization model.

[0033] Step 9: After the network training is completed, the pedestrian re-identification is evaluated and visualized on the same-appearance and cross-appearance test sets of multiple data sets, and the top 1, 5, and 10 hit rates (Rank1, Rank5, Rank10) and the average retrieval accuracy mAP are calculated to prove the promotion of the multi-loss joint optimization method to pedestrian retrieval. In the test phase of the model, the pedestrian's reasoning features in step 4 are selected as the final pedestrian feature vector representation. Compared with high-dimensional features and classification features, the pedestrian's reasoning features balance the optimization goals of multiple losses and have a strong pedestrian representation ability. At the same time, the feature data has a fixed dimension, which is convenient for distance similarity matrix calculation.

[0034] The present invention designs a data sampling algorithm and loss function based on the appearance attributes of pedestrians, and adopts a normalization structure to jointly optimize multiple losses. First, batch images of multiple appearances and multiple identities are sampled using the identity and appearance category of the pedestrians to ensure the stability of model training. Secondly, based on the fixed composition structure of the batch images, the present invention designs a multi-appearance ternary loss and an appearance classification loss, and uses a batch normalization structure to jointly optimize multiple losses, so that the model can acquire the ability to learn the differences between different appearances of the same pedestrian while learning to distinguish different pedestrians, and finally improve the comprehensive retrieval ability of the model under the same appearance and cross-appearance conditions.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The appearance-based sampling strategy used in this invention takes into account the diversity of pedestrian posture, identity, appearance, etc., thereby promoting the stability of cross-appearance pedestrian re-identification network training;

[0037] 2. The multi-appearance ternary loss and appearance attribute loss based on appearance information used in the present invention improve the model's learning ability for pedestrians with different appearances and enhance the retrieval ability;

[0038] 3. The multi-loss joint optimization method used in the present invention balances the optimization objectives of loss in different feature spaces, reduces training fluctuations, and improves feature robustness;

[0039] 4. Compared with other pedestrian re-identification optimization methods, the present invention does not require additional prior information and can significantly improve the model accuracy with only a small computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of an embodiment of the present invention;

[0041] Figure 2 is a partial display diagram of a batch image generated by a sampling algorithm used in an embodiment of the present invention;

[0042] Figure 3 It is a graph of the top ten retrieval results of some pedestrians on the NKUP+ cross-appearance subset of the benchmark network (Densenet121) of the embodiment of the present invention;

[0043] Figure 4 This is a graph of the top ten retrieval results of some pedestrians on the NKUP+ cross-appearance subset using the multi-loss joint optimization model (MLUNet) of an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] The embodiment of the present invention provides a joint optimization method for pedestrian re-identification by integrating appearance information, such as Figure 1 As shown, it includes the following steps:

[0046] Step 1: Preprocessing the cross-appearance person re-identification dataset. The processing steps of the training set images are as follows: 1) resize the image to the network input size (256*128); 2) randomly flip the image horizontally with a probability of 50%; 3) fill 10 pixels with a value of 0 around the image; 4) randomly crop an image of the network input size (256*128) from the image; 5) subtract the mean and divide the variance of the image to normalize the input image, using the mean (0.485, 0.456, 0.406) and variance (0.229, 0.224, 0.225) of the image in ImageNet; 6) randomly erase the area of ​​2% to 40% of the area in the image with a probability of 50%. For the images in the test set, only image scaling and mean and variance subtraction operations are performed: 1) the image size is adjusted to the network input size (256*128), and 2) the image is subjected to mean and variance subtraction to normalize the input image, using the mean (0.485, 0.456, 0.406) and variance (0.229, 0.224, 0.225) of the images in ImageNet.

[0047] Step 2: Based on the identity and appearance attributes of the pedestrians in the training set images, a random sampling algorithm is used to generate batches of images, where each batch consists of 8 pedestrians, each pedestrian contains 2 appearances, and each appearance contains 4 images. Specifically, each batch first randomly selects 8 pedestrians, then selects 2 appearances from the images of each pedestrian, and then selects 4 images from each appearance. When the number of appearances of a pedestrian is less than 2 or the number of images of the appearance is less than 4, the pedestrian or appearance will be removed before the end of this round of iterative training, and these images will be reset at the beginning of each new training iteration.

[0048] Step 3: Use the feature extraction backbone network (Densenet121) trained with the public dataset (ImageNet) to extract the high-dimensional feature matrix of pedestrians, and obtain the pedestrian high-dimensional features f based on generalized mean pooling (GeM Pooling) G :

[0049] f G =Gempool(Backbone(I)).

[0050] Step 4: Use batch normalization to obtain pedestrian inference features f for pedestrian high-dimensional features I :

[0051] f I =BatchNorm(f G ).

[0052] Step 5: Use the fully connected layer (FC) to obtain the identity classification feature f ID , and then use another fully connected layer and batch normalization to obtain the appearance classification feature f APID :

[0053] f ID =FC ID (f I )

[0054] f APID =BatchNorm(FC APID (f I )).

[0055] Step 6: The network model calculates end-to-end: based on the pedestrian high-dimensional features f G Calculate the hardest triplet loss and multi-appearance triplet loss based on the identity classification feature f ID and appearance classification feature f APID Calculate the identity and appearance classification losses and sum them weightedly:

[0056] L=L HardTri (f G )+L ID (f APID )+λ1L MATri (f G )+λ2L AP (f APID ).

[0057] Among them, λ1 and λ2 represent the weight parameters of multi-appearance triplet loss and appearance classification loss, respectively, and both λ1 and λ2 are set to 0.2.

[0058] Based on the fixed identity batch sampling strategy, the hardest triplet loss function is used to reduce the distance between images of pedestrians with the same identity and expand the distance between images of pedestrians with different identities. HardTri :

[0059]

[0060] Among them, α represents the interval parameter, D represents the distance metric, represents the kth image of the pth person in the batch High-dimensional features, 1≤p≤P, 1≤k≤K; k′ represents the k′th image, p′ represents the p′th person;

[0061] Based on the fixed appearance batch sampling strategy, the multi-appearance ternary loss can be used to guide the network to learn the distance between images of the same pedestrian with different appearances. Specifically, it limits the distance between the most difficult negative samples with different appearances by the distance between the most difficult positive samples with the same appearance of the same pedestrian, so that the network learns to distinguish difficult samples with different appearances. MATri :

[0062]

[0063] in, The kth image representing the ath appearance of the pth person in the batch , 1≤p≤P, 1≤a≤A, 1≤k≤K, and pb represents the bth appearance of the pth person.

[0064] Identity classification loss L ID :

[0065]

[0066] Appearance classification loss L AP :

[0067]

[0068] Among them, x i (1≤i≤N), y i 、z i denotes the image and its identity category and appearance category respectively, p(y i |x i ) and p(z i |x i ) represent the image x i Recognized by the model as identity category y i and appearance category z i probability.

[0069] Step 7: Back propagate the gradient of the loss layer to update the weight parameters of the feature extraction backbone network model and the fully connected layer.

[0070] Step 8: Repeat steps 2 to 7 to optimize the multi-loss joint optimization model on the dataset for 120 rounds, with the initial learning rate of the network being 3.5×10 -6 In the first 10 epochs, the network learning rate will increase linearly to 3.5×10 -4 , Subsequently, the learning rate will decay to 0.1 times the current value in 31, 61, and 91 rounds respectively to fine-tune the network weights.

[0071] Step 9: After network training, the query image and candidate image of the test set are input into the multi-loss joint optimization model for forward propagation, and the pedestrian inference feature f is used I As the final pedestrian feature vector representation, the feature f of the query image is then calculated q and the feature f of the candidate image g The Euclidean distance d between q，g =||f q -f g ||2. The smaller the distance between the query image and the candidate image, the higher the similarity. Calculate the distance between each query image and all candidate images and sort them from large to small according to the similarity to obtain a sorted list. Finally, calculate the top k hit rate Rank-k and the average retrieval precision mAP. Comparative experiments are conducted on the NKUP+ and PRCC datasets to prove the effectiveness of this joint optimization method.

[0072] Figure 2 Shows some batches of pedestrian images generated by a random sampling strategy based on identity and appearance. Each row shows all images of a pedestrian in the batch, and the same image bounding box represents the same appearance. Figure 2 There are four appearances in total, with 4 images for each appearance. Figure 2 It can be seen that the sampling algorithm takes into account the differences in pedestrian posture, lighting, background, etc., and especially ensures that the batch contains a diversity of pedestrians and a diversity of appearance, thereby promoting the network's ability to learn cross-appearance features.

[0073] Figure 3 and Figure 4 The results of some pedestrian re-identification of the baseline model and the multi-loss joint optimization model on the NKUP+ cross-appearance subset are shown. The top ten retrieval results of a pedestrian to be retrieved are shown in each row. The leftmost image represents the retrieval image, and the ten query images on the right are arranged from high to low according to the similarity. The black and gray bounding boxes represent the correct and incorrect retrieval results respectively. As can be seen from the figure, the baseline network model has extremely poor retrieval ability for cross-appearance pedestrians and is basically unable to retrieve the target pedestrian. After adopting the joint optimization method for pedestrian re-identification that integrates appearance information, the model's retrieval ability for pedestrians of different appearances has been significantly improved, which confirms that after integrating appearance information, this joint optimization method can improve the performance of the pedestrian re-identification model.

[0074] Table 1 Comparison of retrieval indicators of each network in PRCC dataset

[0075]

[0076] Table 2 Comparison of retrieval indicators of each network in NKUP+ dataset

[0077]

[0078] Tables 1 and 2 quantitatively list the experimental Rank-k and mAP indicators, which are two important evaluation criteria in the field of pedestrian re-identification. In the PRCC dataset with a relatively small number of images and little appearance change, the multi-loss joint optimization model extracts features that improve the Rank1 value of 1.4% / 28.2% and mAP accuracy of 3.8% / 23.9% on the same / cross-appearance subsets respectively; while in the NKUP+ dataset with a large number of images and obvious appearance changes, the multi-loss joint optimization model improves the Rank1 value of 7.5% / 6.8% and mAP of 8.6% / 2.2% on the same / cross-appearance subsets respectively, proving the generalization ability of the multi-loss joint optimization feature.

[0079] The present invention has been described in detail above through embodiments, but the contents described are only exemplary embodiments of the present invention and cannot be considered to limit the scope of implementation of the present invention. The protection scope of the present invention is defined by the claims. Anyone who utilizes the technical solution described in the present invention, or a technician in the field, inspired by the technical solution of the present invention, designs a similar technical solution within the essence and protection scope of the present invention to achieve the above technical effects, or makes equal changes and improvements to the scope of application, etc., shall still fall within the scope of protection covered by the patent of the present invention.

Claims

1. A joint optimization method for person re-identification integrating appearance information, characterized in that: The following steps are involved: Step 1: Preprocess the cross-appearance person re-identification dataset using data augmentation strategies; Step 2: Generate batch images with multiple identities and appearances based on the identity attributes and appearance attributes of the pedestrians in the preprocessed images; Step 3: Use the feature extraction network to extract the high-dimensional feature matrix and perform pooling to obtain the high-dimensional features of pedestrians; Step 4: Perform batch normalization based on pedestrian high-dimensional features to obtain pedestrian inference features; Step 5: Connect different fully connected layers based on pedestrian inference features to obtain identity classification features and appearance classification features respectively; Step 6: Calculate the most difficult triplet loss and multi-appearance triplet loss based on the pedestrian high-dimensional features, and calculate the identity classification loss and appearance classification loss based on the identity classification features and appearance classification features respectively, then weighted sum the four losses and jointly optimize them end-to-end; Step 7: Back propagate the gradient of the loss layer to update the weight parameters of the feature extraction network and the fully connected layer; Step 8: Repeat steps 2-7 until the feature extraction network model converges or the maximum number of iterations is reached to obtain a multi-loss joint optimization model.

2. The joint optimization method for person re-identification integrating appearance information as claimed in claim 1, characterized in that: The following steps are involved: In step 1, the images of the training set are preprocessed using scaling, random horizontal flipping, padding, random cropping, mean subtraction and variance division, and random erasing.

3. The joint optimization method for person re-identification integrating appearance information as claimed in claim 1, characterized in that: In step 2, each batch of images contains a fixed number of pedestrians, each pedestrian contains a fixed number of appearances, and each appearance contains a fixed number of images.

4. The joint optimization method for person re-identification integrating appearance information as claimed in claim 1, characterized in that: The following steps are involved: In step 6, the most difficult ternary loss L HardTri : Among them, α represents the interval parameter, D represents the distance metric, represents the kth image of the pth person in the batch High-dimensional features, 1≤p≤P, 1≤k≤K; k′ represents the k′th image, p′ represents the p′th person; Multi-appearance ternary loss L MATri : in, The kth image representing the ath appearance of the pth person in the batch , 1≤p≤P, 1≤a≤A, 1≤k≤K, and pb represents the bth appearance of the pth person.

5. The joint optimization method for person re-identification integrating appearance information as claimed in claim 1, characterized in that: The following steps are involved: In step 6, the identity classification loss L ID : Appearance classification loss L AP : Among them, x i (1≤i≤N), y i 、z i denotes the image and its identity category and appearance category respectively, p(y i |x i ) and p(z i |x i ) represent the image x i Recognized by the model as identity category y i and appearance category z i probability.

6. The joint optimization method for person re-identification integrating appearance information as claimed in claim 1, characterized in that: The following steps are involved: In step 6, the overall loss L is: L=L HardTri +L ID +λ1L MATri +λ2L AP Among them, λ1 and λ2 represent the weight parameters of multi-appearance triplet loss and appearance classification loss respectively, and L HardTri represents the most difficult ternary loss, L MATri represents the multi-appearance triplet loss, L ID represents the identity classification loss, L AP represents the appearance classification loss.

7. The joint optimization method for person re-identification integrating appearance information as claimed in claim 6, characterized in that: The following steps are involved: Both λ1 and λ2 are set to 0.2.

Images