A pedestrian re-identification method combining style transformation and pose generation

Abstract

The application provides a pedestrian re-identification method combining style transformation and posture generation, comprising the following steps: step one, inputting an image original data set into a data generation module to obtain unified posture images of different styles; step two, inputting the unified posture images of different styles into a classification and recognition module to perform classification and recognition; step three, inputting a query image, extracting image features, and performing similarity measurement on the image features in an image library, and then completing pedestrian re-identification; the pedestrian re-identification method combining style transformation and posture generation simultaneously considers relevant factors of style and posture, eliminates differences between cameras and changes between postures, and learns a more generalized pedestrian re-identification network. In order to simultaneously learn identity-sensitive and visual-sensitive information, this paper combines style and posture information, generates an expanded training data set, simultaneously retains the two, has diverse information, learns more robust pedestrian features, and enhances the generalization ability of the network.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a pedestrian re-identification method combining style transformation and pose generation. BACKGROUND

[0002] As an important branch of intelligent monitoring network technology, pedestrian re-identification technology has attracted more and more attention and research. Pedestrian re-identification is a cross-camera retrieval task, that is, given a pedestrian to be queried, its target is to retrieve the same pedestrian from a database collected from multiple cameras. Pedestrian re-identification technology is the basis of the fields of security, pedestrian and vehicle tracking, and is an important link in building a safe city and a smart city.

[0003] However, due to the collection of images by multiple cameras, different resolutions and environmental light and angle differences will change the appearance of pedestrians and the background, resulting in the same identity being identified as different pedestrians due to appearance changes, and different identities being identified as the same pedestrian due to the same pose. In the face of the challenges of low discriminability and poor robustness of pedestrian features caused by camera resolution, light, background occlusion and pose changes, in order to eliminate the influence of these challenges, it is necessary to further study new pedestrian re-identification methods.

[0004] In view of the problem that different cameras change continuously in actual application and the poses of pedestrians in training data differ, most of the existing researches are based on generative adversarial network (GAN) learning to learn stable feature representation with invariance under different cameras. GAN is used in pedestrian re-identification to perform data augmentation, mainly using GAN to solve the problem of insufficient data in pedestrian re-identification. Through a pose conversion framework, the pose (i.e. skeleton) is extracted from a data set MARS with rich human pose changes, and then samples are generated in a new pose based on a GAN model, eliminating the influence of pose on the appearance of people. CamStyle is used to perform data enhancement from the perspective of camera style adaptation, smoothing the differences in camera style to reduce the influence of overfitting of the convolutional network.

[0005] Although the existing typical method solves the problems of data lack and camera style difference or pose change through GAN, the data generated by GAN lacks diversity, and the trained model does not have good enough generalization ability. And when eliminating the influence of pose on the appearance of people alone, the changes in appearance and background caused by the existence of camera changes are ignored; when eliminating the camera style difference, the potential positive sample pairs in the target domain are ignored, which may cause the pedestrian re-identification model to be sensitive to other features in the target domain, such as pose and background changes. SUMMARY

[0006] In order to better solve the existing pedestrian recognition method, the present application designs and proposes a pedestrian re-identification method combining style transformation and posture generation, comprising the following steps:

[0007] Step one, input the image original data set into the data generation module, and obtain the unified posture image of different styles;

[0008] Step two, input the unified posture image of different styles into the classification recognition module for classification recognition;

[0009] Step three, input the query image, extract the image features, and perform similarity measurement with the image features in the image library, and then complete the pedestrian re-identification.

[0010] Further, the specific process of the step one, inputting the image original data set into the data generation module and obtaining the unified posture image of different styles is:

[0011] S201, input the image original data set into the style transfer module, output multiple images with different camera styles through style transfer learning;

[0012] S202, input the image original data set into the posture estimation module, perform posture estimation through AlphaPose, and extract 8 standard posture skeletons;

[0013] S203, input the images with different styles and the posture skeletons into the DCGAN network, and output the unified posture images with different styles.

[0014] Further, the specific process of the step two, inputting the unified posture image of different styles into the classification recognition module,

[0015] Classification recognition is:

[0016] S301, train the classification recognition network based on the expanded training data set, and after training, remove the full connection layer and the classifier of the network model;

[0017] S302, input all images in the pedestrian library as network input, and output before the full connection layer,

[0018] The output is the pedestrian feature.

[0019] Further, the specific process of the step three, inputting the query image, extracting the image features, and performing similarity measurement with the image features in the image library, and then completing the pedestrian re-identification is:

[0020] S401, input the query image as the classification recognition network, and extract the image features;

[0021] S402, using the Euclidean distance to calculate the similarity between the query image and the image features in the image library, judging whether there is a pedestrian in the query image according to the similarity, and completing the pedestrian re-identification task.

[0022] The advantage of the present application is that the pedestrian re-identification method combining style transformation and pose generation is provided, the related factors of style and pose are considered, the differences between cameras and the changes between poses are eliminated, and a more generalized pedestrian re-identification network is learned. In order to be able to learn identity-sensitive and vision-sensitive information at the same time, the style and pose information are combined in this paper, the generated expanded training data set retains both at the same time, has diversity information, learns more robust pedestrian features, and enhances the generalization ability of the network. First, through style transfer, the generated image learns the style of different cameras, reduces the difference between cameras; then the pose estimation is introduced, the different style images have the same pose, and the influence of the pose change is eliminated; finally, random erasing is introduced for data enhancement, and the partial occlusion problem is solved. The network is trained using the newly synthesized data, the pedestrian re-identification network is optimized by fine-tuning the ResNet-50 network structure, the performance of pedestrian re-identification is improved, and finally a pedestrian re-identification method combining style and pose is designed.

[0023] The present application will be described in detail below in combination with the drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is the overall framework diagram of the pedestrian re-identification method combining style transformation and pose generation.

[0025] Figure 2 is the pose skeleton extraction schematic diagram.

[0026] Figure 3 is the training result schematic diagram of different random erasing probabilities P.

[0027] Figure 4 is the training result schematic diagram on the Market-1501 data set.

[0028] Figure 5 is the training result schematic diagram on the DukeMTMC-reID data set. DETAILED DESCRIPTION

[0029] In order to further illustrate the technical means and effects taken by the present application to achieve the predetermined purpose, the specific implementation mode, structural features and effects of the present application are described in detail below in combination with the drawings and examples.

[0030] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be clearly and completely described in the description of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of the present application.

[0031] In the description of the present application, it should be understood that the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "alignment", "overlap", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0032] The terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features; in the description of the present application, unless otherwise specified, the meaning of "multiple" is two or more.

[0033] Embodiment 1

[0034] The present example provides a pedestrian re-identification method combining style transformation and pose generation as shown in Figure 1 The method mainly includes two network modules, an image data generation module for generating multiple style unified poses, and a classification and recognition network module.

[0035] The first part of the image data generation module is composed of style transfer learning and pose estimation, which are jointly used as the input of the generation network, and the output of the generation network is used as the input of the second part of the classification and recognition network. The image features are extracted, classified and recognized through the full connection layer FC-T, T is the pedestrian identity category in the data set, the full connection layer is removed, the query image is input, the image features are extracted, and the similarity between the image features in the image library is measured, and then the pedestrian re-identification is completed. This part is a network structure improved based on ResNet-50.

[0036] The pedestrian re-identification method combining style transformation and pose generation specifically includes the following steps:

[0037] Step 1, input the original image data set into the data generation module to obtain unified pose images of different styles;

[0038] Step two, input different style unified posture image into classification recognition module for classification recognition;

[0039] Step three, input query image, extract image features, and perform similarity measurement on image features in the image library, and then complete pedestrian re-identification.

[0040] Further, the specific process of step one, inputting image original data set into data generation module to obtain unified posture images with different styles is:

[0041] S201, input image original data set into style transfer module, output multiple images with different camera styles through style transfer learning;

[0042] S202, input image original data set into posture estimation module, perform posture estimation through AlphaPose, and extract 8 standard posture skeletons;

[0043] S203, input images with different styles and posture skeletons into DCGAN network, and output unified posture images with different styles.

[0044] Further, the DCGAN network (Deep Convolutional GAN, deep convolutional adversarial network) is a combination of deep convolutional network and GAN. The discriminator and generator in DCGAN use convolutional neural network (CNN) to replace the multi-layer perceptron in GAN, and at the same time, in order to accelerate the convergence speed of the network in training, the fully connected layer behind the convolutional layer in the network is deleted, so that the network becomes a fully convolutional network. In order to make the training process more stable, batch normalization operation is added after the input layer.

[0045] Further, the specific process of step two, inputting unified posture images with different styles into classification recognition module,

[0046] is:

[0047] S301, train the classification recognition network based on the expanded training data set, and after training, remove the fully connected layer and classifier of the network model;

[0048] S302, input all images in the pedestrian image library as network input, and output before the fully connected layer,

[0049] The output is pedestrian features.

[0050] Further, the specific process of step three, inputting query image, extracting image features, and performing similarity measurement on image features in the image library, and then completing pedestrian re-identification is:

[0051] S401, input the query image into the classification recognition network to extract image features;

[0052] S402, calculate the similarity between the query image and the image features in the image library using the Euclidean distance, and determine whether there is a pedestrian in the query image according to the similarity, thereby completing the pedestrian re-identification task.

[0053] To solve the differences in resolution, environmental light, etc. between different cameras, style transfer is used to eliminate the differences between camera styles. Assuming that the data set used comes from pedestrian images collected under C cameras, for each image in the training set, C-1 images of other camera styles can be generated, and the C images composed of the real image and the C-1 generated images have the same identity and belong to the same pedestrian ID.

[0054] On the data set Market-1501, given a pedestrian image under camera 1, style transfer models under the other 5 cameras are learned through CycleGAN, and the original image and the style transfer model are combined to generate pedestrian images under the styles of the other 5 cameras.

[0055] Pose estimation:

[0056] Pose estimation uses the pose detection tool AlphaPose, that is, given an input image X of pedestrian 1 i , the pose image X pi of the corresponding pose skeleton of pedestrian i can be detected. The pose image X pi generated by the pose estimator and the image X j of pedestrian j are used as input images, and the DCGAN network is used to synthesize an image of pedestrian j in the pose of pedestrian i , that is , which has a different pose but the same identity as X j .

[0057] The process of extracting the skeleton of the pose of a pedestrian detected by AlphaPose is shown in Figure 2 , which takes 18 key points, including the nose, eyes, ears, neck, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles.

[0058] Different style multi-pose generation:

[0059] Pose image generation mainly combines real images with different pose skeletons to generate images with target poses. Unlike the previous method of generating multi-pose images using only real images, this method selects images generated by style transfer and uses them as actual input images together with target poses, so that the generated images not only have different poses but also have style transfer.

[0060] In the process of picture generation, the DCGAN network structure is adopted, in order to accelerate the convergence speed of the network in the training, the full connection layer behind the convolution layer in the network is deleted, so that the network becomes a full convolution network. In order to make the training process more stable, the batch normalization operation is added after the input layer. In order to utilize the residual information, based on the ResNet network architecture, the structure of the automatic encoder encoder-decoder is used, the encoder network gradually downsamples the input pedestrian picture to the bottleneck layer, and then the decoder obtains the output from the encoder, and reversely generates different posture pedestrian pictures. Through the posture image generation network, a plurality of synthetic images with different postures can be generated for the training set, which retains the original pedestrian related information, only changes the posture, so that each real image X i and the image after changing the posture belong to the same identity. Therefore, the posture conversion loss function is introduced, as shown in formula (1):

[0061]

[0062] Wherein, d represents the identity of the real image X j and the image after changing the posture .

[0063] Random erasing:

[0064] Under different cameras, due to different angles, the background part may be blocked. In order to alleviate the problem of decreasing the pedestrian re-identification accuracy caused by this factor and reduce the deviation caused by background occlusion, random erasing data enhancement is introduced in this method, and the input is an original image I, and the output is a picture after data enhancement with various occlusions.

[0065] Given the original image I, its width and height are W and H respectively, and the area is S. Assuming that the random erasing area is I e , the area is S e , the width and height are W e and H e , and the aspect ratio r e is in the range (r1, r2). Set the random erasing probability P, first randomly find a pixel point (x e , y e ) on the original image I, if the random erasing process satisfies formula (2), change the pixel value of all points in the random erasing area I e to a random number in (0, 255), that is, the random erasing operation is completed on the image. Through experimental comparison, when the random erasing probability P=0.5, the best recognition effect can be achieved.

[0066]

[0067] where W e and H e are the width and height of the random erasing region, and W and H are the width and height of the given original image, respectively.

[0068] Classification recognition network:

[0069] Different from the original backbone network, the generated new data is used as training data, the network parameters trained in the ResNet-50 network except the full connection layer are reserved, and the moving step of the first bottleneck residual block convolution kernel of Layer5 in the model is modified to (1, 1), and the full connection layer FC-1000 in the ResNet-50 network model structure is deleted, and a full connection layer FC-1024 with 1024 neurons is used to replace it. Since the training data set contains T different pedestrian identities, the final result of recognition is T classes.

[0070] In summary, this pedestrian re-identification method combining style transformation and pose generation considers the relevant factors of style and pose, eliminates the differences between cameras and the changes between poses, and learns a more generalized pedestrian re-identification network. In order to be able to learn identity-sensitive and visual-sensitive information at the same time, this paper combines style and pose information, and the generated expanded training data set retains both, has diverse information, learns more robust pedestrian features, and enhances the generalization ability of the network. First, through style transfer, the generated images learn the styles of different cameras, reducing the differences between cameras; then introduce pose estimation, generate different style images with the same pose, eliminate the influence of pose changes; finally, introduce random erasing for data augmentation, solve the problem of partial occlusion. Use the newly synthesized data to train the network, optimize the pedestrian re-identification network by fine-tuning the ResNet-50 network structure, improve the performance of pedestrian re-identification, and finally design a pedestrian re-identification method combining style and pose.

[0071] Example 2

[0072] The implementation framework of the method is PyTorch (1.4.0), the computer processor is Intel(R) Xeon(R) Gold 6254 CPU @ 3.10 GHz, the memory is 16 GB, the graphics card is GeForce_RTX_2080_Ti, and the operating system is Ubuntu 18.04, and the programming language is Python.

[0073] In order to objectively evaluate the performance of the method, the Market-1501 and DukeMTMC-reID benchmark data sets are used for comparison test with other typical methods.

[0074] The main parameter settings in the network training and optimization process are as follows: in the style transfer learning training, the picture size is set to 256*128, the epoch is 50, the learning rate is set to 0.0002 for the generator in the first 30 epochs, the learning rate for the discriminator is 0.0001, and the learning rate is decayed to 0 in the remaining 20 epochs; in the training of the pose conversion network, the epoch is set to 20, the batch size is 32, the learning rate is 0.0002, and the dropout is 0.5; in the training of the pedestrian re-identification network, the epoch is set to 90, the batch size is 128, the learning rate is initially set to 0.01, then scaled by a factor of 10 at each stage, the dropout is 0.5, and the output of the Pooling-5 layer is extracted as the image feature.

[0075] Quantitative evaluation

[0076] The rank-k and mAP indicators are used to evaluate pedestrian re-identification. Rank-k is the selection of the top k pedestrian pictures with the closest similarity to the query picture in the candidate set through similarity ranking, and the probability of the k pedestrian pictures containing the correct picture is calculated. In the experiment, rank-1, rank-5, and rank-10 are used as the final evaluation indicators, and rank-1 is compared with other methods. mAP is the identification of given pedestrian samples in the candidate set, and the identified samples are sorted according to similarity.

[0077] The mAP experimental results of random erasure with different probability settings on the Market-1501 dataset are shown in Figure 3 .

[0078] As can be seen from Figure 3 , when the random erasure probability P is 0.5, the erasure area of the training image is appropriate, and it will not cause the network to extract the same features as the original training picture due to too little erasure area or be unable to train due to too much erasure area, so the recognition effect is best when P is 0.5.

[0079] After the experiment is trained, the results of combining style and pose and introducing random erasure are saved respectively, and the CMC (Cumulative Matching Characteristic Curve) scores are shown in Figure 4 and 5 .

[0080] From Figure 4 and Figure 5It can be seen that in the method, the random erasing data enhancement method is introduced, the original image is randomly erased, part of the pedestrian image in the training is blocked, the network is forced to train on the remaining part, more robust pedestrian features are extracted, the problem of partial occlusion is solved, and the effect of pedestrian re-identification is improved.

[0081] Based on the Market-1501 and DukeMTMC-reID data sets, the improved method is compared with five typical related pedestrian re-identification algorithms IDE, GAN, PN-GAN, CamStyle and RandEra, and the rank-1 and mAP evaluation indexes are analyzed, and the experimental results are shown in table 1 and table 2.

[0082] Table 1 Comparison of performance of different methods on Market1501 data set

[0083]

[0084]

[0085] Table 2 Comparison of performance of different methods on DukeMTMC-reID data set

[0086]

[0087] As shown in table 1, the rank-1 of the network model trained only using the original model ResNet50 network on the Market1501 data set is only 72.54, and the mAP is 46.00. After the improved ResNet50 model, the optimization of training parameters and training data, the rank-1 of the network model of the method is 89.42%, and the mAP is 72.64%. Compared with the rank-1 and mAP of the network model using only style or posture before improvement, the rank-1 and mAP of the method are improved, which fully shows that the training method combining style and posture is feasible. Because only using style change can only smooth the difference between cameras, the other information of pedestrians in the benchmark data set is not rich, so considering the posture related information at the same time can make the trained network not too dependent on the posture change of pedestrians to distinguish pedestrians and extract more robust pedestrian features. On this basis, the random erasing data enhancement method is further introduced, and it can be seen that the values of rank-1 and mAP continue to improve. Compared with the five typical methods, it can be concluded that the method in the paper has led the five methods in the rank-1 and mAP indexes, which shows that rich sample information is helpful to improve the performance of pedestrian re-identification method.

[0088] From Table 2, it can be concluded that the rank-1 of the improved network model of the method on the DukeMTMC-reID dataset reaches 79.00%, and the mAP reaches 58.40%. The quantitative evaluation results of the method are slightly higher than those of other methods, which shows that the method of combining style and posture and the introduction of random erasing data enhancement method are feasible. By considering the influence of posture change while style transfer, converting the original picture into a picture of other camera style and converting the posture into a unified posture, the camera change and the contingency of recognizing pedestrians according to the posture change can be eliminated, more representative pedestrian features can be learned, and the introduction of random erasing can make the trained network have better generalization ability when processing occluded images. Therefore, the method has certain feasibility in improving the accuracy of pedestrian re-identification.

[0089] From the above quantitative evaluation results, it can be concluded that the pedestrian re-identification method combining style and posture generated in the present application can improve the diversity of the dataset to a certain extent, and can effectively solve the problem of low discriminability and poor robustness of pedestrian features caused by changes such as light, background partial occlusion and posture between datasets.

[0090] The above is a further detailed description of the present application in combination with specific preferred embodiments, and cannot be regarded as limiting the specific implementation of the present application to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, a number of simple deductions or substitutions can be made without departing from the concept of the present application, and all of them should be regarded as falling within the protection scope of the present application.

Claims

1.A pedestrian re-identification method combining style transformation and pose generation, characterized in that, Comprise the following steps: Step one, the image original data set input data generation module, get different style of unified posture image; The specific process is: S201, input image original data set to style transfer module, through style transfer learning, output multiple images with different camera styles; S202, input image original data set to posture estimation module, through AlphaPose posture estimation, extract 8 standard posture skeleton; S203, different style of image and posture skeleton input DCGAN network, output unified posture image with different style; Step two, the different style of unified posture image input classification recognition module, classification recognition; The specific process is: S301, based on the training data set of expansion training classification recognition network, after training, remove the full connection layer and classifier of network model; S302, all images in pedestrian library as network input, output before full connection layer, output is pedestrian feature; Step three, input query image, extract image feature, and image feature in image library similarity measurement, and then complete pedestrian re identification; The specific process is: S401, the query image as classification recognition network input, extract image feature; S402, using Euclidean distance to calculate the similarity between the query image and the image feature in the image library, according to the similarity to judge whether there is the pedestrian in the query image, complete the pedestrian re identification task.

Images