Pedestrian re-identification method based on three-dimensional reconstruction and image generation

Abstract

The invention provides a pedestrian re-identification method based on three-dimensional reconstruction and image generation, and belongs to the technical field of machine vision. According to the invention, pedestrian images in an original pedestrian image set are utilized to generate a corresponding three-dimensional human body model; selecting a specific angle of the three-dimensional human body model to generate a two-dimensional pedestrian rendering graph; generating a virtual pedestrian image in combination with the two-dimensional pedestrian rendering graph and the original pedestrian image; adding the virtual pedestrian image into the original pedestrian image set to construct an enhanced image set; training a pedestrian re-identification model based on the enhanced image set; and inputting a to-be-detected pedestrian image to the trained pedestrian re-identification model to obtain a pedestrian identification result. According to the invention, the original pedestrian image set is enriched by generating images of different angles of portraits in the original pedestrian image set, and the problem of scarcity of pedestrian re-identification data is solved; and the model is trained and updated on the enhanced data set in combination with classification loss and triple loss, so that the model can correctly learn the generated image information, and the pedestrian re-identification precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing and machine vision technology, and in particular to a pedestrian re-identification method based on three-dimensional reconstruction and image generation. Background Art

[0002] Pedestrian re-identification technology is widely used in many fields, including security monitoring, retail, intelligent transportation, urban planning, and event safety management.

[0003] Existing person re-identification methods mainly rely on extracting features from two-dimensional images for matching, which requires a large amount of labeled pedestrian image data for model training. However, in actual scenarios, there are often problems with small amounts of training data and uneven distribution of training data, especially data from different camera perspectives or in different environments, which may have a large distribution offset. Therefore, in complex scenarios (such as perspective changes, occlusion, lighting differences, etc.), the performance of the model is significantly reduced, resulting in insufficient accuracy of person re-identification results.

[0004] Therefore, how to improve the accuracy and robustness of pedestrian re-identification when data is scarce and unevenly distributed is a technical problem that needs to be solved. Summary of the invention

[0005] In view of this, the present invention provides a pedestrian re-identification method based on three-dimensional reconstruction and image generation, which utilizes the three-dimensional reconstruction and image generation methods to generate more virtual pedestrian images on the basis of the original pedestrian image set, constructs an enhanced image set, and trains a pedestrian re-identification model based on the enhanced image set, so as to improve the accuracy of pedestrian re-identification when the data amount is insufficient and the data distribution is uneven.

[0006] To this end, the present invention provides the following technical solutions: A pedestrian re-identification method based on three-dimensional reconstruction and image generation, comprising: Generate a 3D human body model using pedestrian images in an original pedestrian image set; Rotating the three-dimensional human body model to a preset angle to generate a two-dimensional pedestrian rendering; Generate a virtual pedestrian image by combining the two-dimensional pedestrian rendering and the original pedestrian image; The virtual pedestrian images are added to the original pedestrian image set to construct an enhanced image set; Combining classification loss and triplet loss to train the person re-identification model on the augmented image set; Input the pedestrian image to be detected into the trained pedestrian re-identification model to obtain the pedestrian recognition result.

[0007] Furthermore, the generating of the three-dimensional human body model using the pedestrian images in the original pedestrian image set includes: Extract features from the original pedestrian image I through a convolutional neural network ; Utilize the features through a multi-body linear model based on skin vertices Generate a 3D human body model:

[0008] in, and They are shape parameters and posture parameters respectively.

[0009] Furthermore, the generating of the virtual pedestrian image by combining the two-dimensional pedestrian rendering image and the original pedestrian image includes: Extract identity features from original pedestrian images; Extract posture features through 2D pedestrian renderings; Recombining identity features and posture features to generate virtual pedestrian images.

[0010] Furthermore, the identity feature and the posture feature are recombined by feature decoupling and dual generation network to generate a virtual pedestrian image, including: The generator generates a generated image that combines identity features and posture features; The discriminator distinguishes between real images and generated images; The generator and the discriminator are trained adversarially to optimize the generated image, and the optimized generated image is output as a virtual image.

[0011] Further, the generator includes: an identity generator and an attribute generator; The identity generator is responsible for maintaining the identity information of the pedestrian image; The attribute generator combines different pose features to change the pose of the generated image.

[0012] Furthermore, the pedestrian re-identification model includes: The global and local features of pedestrian images in the enhanced image set are extracted through the backbone network; The global features and local features are reduced in dimension and used for classification tasks and metric learning tasks respectively.

[0013] Furthermore, the classification task is optimized by the classification loss.

[0014] Furthermore, the metric learning task is optimized by the triplet loss.

[0015] Advantages and positive effects of the present invention: The present invention generates multi-angle pedestrian images to construct an enhanced image set, and trains a pedestrian re-identification model based on the enhanced image set, thereby solving the problem of scarce pedestrian re-identification data, thereby improving the accuracy and robustness of pedestrian re-identification; the present invention extracts image features and constructs a three-dimensional human model through a score-guided human mesh recovery method, thereby improving the generation quality of virtual pedestrian images; the present invention combines classification loss and triplet loss to train and update the model on the enhanced data set, thereby ensuring that the model can correctly learn the generated image information, optimizing the model under the condition of uneven data distribution, and improving the accuracy of pedestrian re-identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0017] Figure 1 This is a flow chart of a method according to an embodiment of the present invention; Figure 2 It is a simulation experiment indicator change trend diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0019] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] The present invention provides a pedestrian re-identification method based on three-dimensional reconstruction and image generation. First, the original pedestrian images are used to generate virtual pedestrian images to expand the public data set of pedestrian re-identification to achieve data enhancement: the original pedestrian images of the public data set of pedestrian re-identification are three-dimensionally modeled, and these three-dimensional human body models are rotated at multiple angles, and the two-dimensional projections of the three-dimensional human body models at specific angles are selected as new structures to simulate the postures of pedestrians in different directions; then the new angle structure is combined with the original pedestrian images to generate virtual pedestrian images to achieve data enhancement. Then the virtual pedestrian images are merged into the original pedestrian image set to form a new training data set with richer data. Finally, the model is trained and updated on the training data set: the classification loss and triplet loss are combined to train and update the model on the training data set to ensure that the model can correctly learn the features of the enhanced image, thereby solving the problem of insufficient data for pedestrian re-identification and improving the accuracy of pedestrian re-identification.

[0021] Combination Figure 1 Yes, the method of the present invention further illustrates: S1. Generate a 3D human model from a single original pedestrian image, and select a specific angle of the 3D human model to generate a 2D pedestrian rendering.

[0022] In this embodiment, the Score-Guided Human Mesh Recovery (ScoreHMR) method is used to extract the postures of people in the original pedestrian image set to generate a three-dimensional human model.

[0023] Specifically, features are extracted from the input image I through a convolutional neural network , and the features Passed to the SMPL model (Skinned Multi-Person Linear Model) to generate a 3D mesh , that is, a three-dimensional human body model:

[0024] in, and are the shape and pose parameters respectively.

[0025] A scoring mechanism is introduced to optimize the generation process of the 3D human model and gradually adjust the parameters of the 3D mesh, including posture, shape, and perspective, so that the generated 3D human model is more consistent with the pedestrians in the image.

[0026] The 3D human body model is rotated at a specific angle and rendered to generate a 2D pedestrian rendering.

[0027] S2. Generate a virtual pedestrian image using the two-dimensional pedestrian rendering and the original pedestrian image to achieve data enhancement.

[0028] The identity features are extracted from the original pedestrian image; the posture features are extracted from the two-dimensional pedestrian rendering image; the identity features and the posture features are recombined to generate a virtual image.

[0029] In this embodiment, virtual image generation is performed through feature decoupling and dual generation networks, including: The generator generates images that match the target posture and identity features, where the identity generator is responsible for maintaining the identity information of the original pedestrian image; the attribute generator combines different posture features to change the posture of the generated image.

[0030] The generated images are optimized through the generative adversarial network to improve their realism, including: Through adversarial training between the generator and the discriminator, the generator generates images that match the target posture and identity characteristics, making the generated images as realistic as possible; the discriminator distinguishes between real images and generated images, and improves the generation quality by continuously providing feedback to the generator. The formula is expressed as:

[0031] Among them, x is the real image, z is the input noise of the generator, is the image generated by the generator, is the probability that the discriminator determines that image x is real; represents mathematical expectation.

[0032] During the adversarial training process, the model uses multiple loss functions, including identity consistency loss, reconstruction loss, and style consistency loss, to ensure that the generated images are consistent with the original pedestrian images in details and style.

[0033] S3, merging the generated virtual pedestrian image into the original pedestrian image set to construct an enhanced image set; In this embodiment, the rule for merging the generated virtual pedestrian image into the original pedestrian image set is to increase the numerical part representing the number of frames of the character image in the file name, and traverse all images to increase this part of the number until a non-conflicting name is found so that it does not repeat with existing files.

[0034] S4. Build a deep learning network, train and update the model on the enhanced image set.

[0035] In this embodiment, the training process uses ResNet-50 as the backbone network, which includes three branches for extracting global and local features.

[0036] The training process includes the following steps: Feature extraction: Image features are extracted through the backbone network and processed through three branches. The global branch extracts overall features, and the two local branches extract local features respectively, and process them through different segmentation strategies.

[0037] Feature processing: The features of the global and local branches are used for classification tasks and metric learning tasks respectively after dimension reduction.

[0038] The classification task is optimized by Softmax Loss, and the metric learning task is optimized by triplet loss. The loss function formula is as follows:

[0039]

[0040] Where: N represents the total number of training samples, C represents the total number of categories, i represents the index of the i-th sample, represents the true category index of the i-th sample, represents the feature vector of the i-th sample, represents the weight vector of category K, represents the bias term for category k, and Indicates the true category of the corresponding sample The weight vector and bias term of ; P represents the number of sampled pedestrian identity categories, K represents the number of images sampled for each pedestrian identity, a represents the sample index of the anchor point, p represents the sample index of the positive sample, and n represents the sample index of the negative sample. The feature vector representing the anchor sample a of identity i, The feature vector representing the anchor sample p of identity i, The feature vector representing the negative sample n of identity j (j≠i), Indicates the minimum interval between positive and negative samples.

[0041] Joint optimization: By jointly optimizing two loss functions, Softmax Loss is used to improve feature discrimination, and triplet loss optimizes feature sorting performance, ultimately generating feature representations for person re-identification.

[0042] The effect of the method of the present invention is further illustrated by the following simulation experiment: The experimental dataset is the pedestrian re-identification dataset Market1501; 1) Input image size: The input image is resized to a fixed size of 384×128 to capture more detailed information about pedestrians.

[0043] 2) Pre-trained weights: The ResNet-50 backbone network is initialized using weights pre-trained on the LUP dataset.

[0044] 3) Batch sampling strategy: Mini-batch structure: randomly select P identities from the training set, and randomly sample K images for each identity. Set P=16, K=4, that is, each mini-batch contains P×K =64 images. Satisfy the sample selection requirements of triplet loss and ensure that each batch of data contains enough positive and negative samples.

[0045] 4) Dataset setting: Use the Market1501 dataset for image generation, select a 2D pedestrian rendering image with the character model in the original image rotated 45° horizontally, merge the generated image into the original dataset, and use the new generated dataset for model training.

[0046] 5) Evaluation indicators: Rank-k accuracy: indicates whether the correct matching result appears in the first k candidate results returned; mAP: The average of all mean precisions (APs), which measures the quality of result ranking in retrieval tasks.

[0047] During the training process, an evaluation is performed every 50 rounds, by Figure 2 It can be seen that all indicators are constantly rising and approaching unchanged after 350 rounds, so the total number of training rounds is selected as 400 epochs.

[0048] The method proposed in the present invention is compared with other pedestrian re-identification methods under the same conditions. As shown in Table 1, the Rank-1, Rank-5, Rank-10 and mAP of the method of the present invention are better in the comparison and are better than the previous methods, which can show that the method is effective and improves the accuracy of pedestrian re-identification. Among them, the mAP is significantly improved, indicating that the model has excellent global ranking performance in the retrieval task, can well rank the correct samples to the front, and is reasonably distributed in the entire retrieval list, indicating that the comprehensive performance of the method model of the present invention is good.

[0049] Table 1

[0050] In summary, the present invention constructs a new training data set with richer data through data enhancement, and performs model training on the enhanced data set through joint training, thereby solving the problem of scarce pedestrian re-identification data; ensuring that the model can correctly learn the generated image information, thereby improving the accuracy of pedestrian re-identification.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pedestrian re-identification method based on three-dimensional reconstruction and image generation, characterized in that: include: Generate a 3D human body model using pedestrian images in an original pedestrian image set; Rotating the three-dimensional human body model to a preset angle to generate a two-dimensional pedestrian rendering; Generate a virtual pedestrian image by combining the two-dimensional pedestrian rendering and the original pedestrian image; The virtual pedestrian images are added to the original pedestrian image set to construct an enhanced image set; Combining classification loss and triplet loss to train the person re-identification model on the augmented image set; Input the pedestrian image to be detected into the trained pedestrian re-identification model to obtain the pedestrian recognition result.

2. According to claim 1, a pedestrian re-identification method based on three-dimensional reconstruction and image generation is characterized in that: The method of generating a three-dimensional human body model by using pedestrian images in the original pedestrian image set includes: Extract features from the original pedestrian image I through a convolutional neural network ; Utilize the features through a multi-body linear model based on skin vertices Generate a 3D human body model: in, and They are shape parameters and posture parameters respectively.

3. According to claim 1, a pedestrian re-identification method based on three-dimensional reconstruction and image generation is characterized in that: The step of combining the two-dimensional pedestrian rendering image and the original pedestrian image to generate a virtual pedestrian image includes: Extract identity features from original pedestrian images; Extract posture features through 2D pedestrian renderings; Recombining identity features and posture features to generate virtual pedestrian images.

4. According to claim 3, a pedestrian re-identification method based on three-dimensional reconstruction and image generation is characterized in that: The virtual pedestrian image is generated by recombining the identity feature and the posture feature through feature decoupling and dual generation network, including: The generator generates a generated image that combines identity features and posture features; The discriminator distinguishes between real images and generated images; The generator and the discriminator are trained adversarially to optimize the generated image, and the optimized generated image is output as a virtual image.

5. According to claim 4, a pedestrian re-identification method based on three-dimensional reconstruction and image generation is characterized in that: The generator includes: an identity generator and an attribute generator; The identity generator is responsible for maintaining the identity information of the pedestrian image; The attribute generator combines different pose features to change the pose of the generated image.

6. The pedestrian re-identification method based on three-dimensional reconstruction and image generation according to claim 1, characterized in that: The pedestrian re-identification model includes: The global and local features of pedestrian images in the enhanced image set are extracted through the backbone network; The global features and local features are reduced in dimension and used for classification tasks and metric learning tasks respectively.

7. The pedestrian re-identification method based on three-dimensional reconstruction and image generation according to claim 6, characterized in that: The classification task is optimized by the described classification loss.

8. The pedestrian re-identification method based on three-dimensional reconstruction and image generation according to claim 6, characterized in that: The metric learning task is optimized by the triplet loss.

Images