A Generative Model-Based Method for Cross-Appearance Person Re-identification and Detection

Abstract

This invention provides a cross-appearance pedestrian re-identification and detection method based on a generative model, belonging to the field of pedestrian re-identification technology. The method includes the following steps: In the generative model, body shape features and appearance features are exchanged between different pedestrian images to generate new pedestrian images; the RGB images of the pedestrian images are processed by a pre-trained edge detection network and a pedestrian semantic segmentation network to obtain pedestrian contour sketches and pedestrian resolution maps; the pedestrian contour sketches, RGB images, and pedestrian resolution maps from the pedestrian dataset are input into the same backbone network to extract features, the features are fused, and then inference training is performed. This invention uses a generative model to enhance the appearance dimension of pedestrian images and introduces the generated pedestrian images into the model's training phase; the three modalities use the same backbone network to extract features, which are then fused for inference, guiding the model to learn the key features of different appearances of the same pedestrian. The model exhibits more robust performance in cross-appearance scenarios.

Description

Technical Field

[0001] This invention belongs to the field of pedestrian re-identification technology, and specifically relates to a cross-appearance pedestrian re-identification detection method based on a generative model. Background Technology

[0002] Currently, computer vision technology has a wide range of applications. Among them, facial recognition technology surpasses human capabilities and is widely used in industries such as manufacturing, healthcare, and education. However, the academic and industrial communities are increasingly focusing on topics with greater research and practical value—pedestrian re-identification. In real-world scenarios such as transportation and industrial manufacturing, faces are often blurred or even nonexistent, limiting the effectiveness of facial recognition.

[0003] Pedestrian re-identification (Re-ID) aims to retrieve other images of the same pedestrian across multiple devices, given a monitored pedestrian image. Pedestrian re-identification technology can compensate for the failure of facial recognition and the visual limitations of fixed cameras, and can be applied in video surveillance, smart security, and smart living. Because Re-ID requires finding the target in images and videos from multiple devices, the resolution and location of the devices vary, the coverage of the devices does not overlap, resulting in a lack of coherent information, and there are differences in scene lighting, background, and occlusion. Furthermore, the target's pose and appearance can change, all of which pose significant challenges to pedestrian re-identification technology.

[0004] Deep learning-based feature learning methods for person re-identification. Global features: This primarily utilizes the entire body image for feature learning, with common improvements including attention mechanisms and multi-scale fusion. Local features: This leverages local image regions, such as partial structures of the pedestrian or simple vertical region divisions, for feature learning. These local features are then aggregated into final pedestrian features for recognition. Auxiliary features: This utilizes auxiliary information to enhance feature learning, such as semantic information, viewpoint information, domain information, information generated by GANs, and data augmentation. Specific network design: Leveraging the characteristics of the Re-ID task, fine-grained, multi-scale network structures are designed to better suit Re-ID scenarios.

[0005] Deep learning-based metric learning-based person re-identification methods mainly include the design of different types of loss functions and improvements to sampling strategies. Identity loss: Treating the Re-ID training process as an image classification problem, different images of the same pedestrian are treated as one category; common methods include the Softmax cross-entropy loss function. Validation loss: Treating the Re-ID training as an image matching problem, binary classification is used to determine whether images belong to the same pedestrian; common methods include contrastive loss functions and binary classification loss functions. Triple loss: Treating the Re-ID training as an image retrieval problem, the feature distance between images of the same pedestrian should be less than the feature distance between images of different pedestrians, and various improvements thereof. Improvements to training strategies include adaptive sampling methods and different weight allocation strategies.

[0006] Cross-appearance person re-identification methods: With the release of cross-appearance person re-identification datasets, researchers have proposed cross-appearance person re-identification methods for these datasets, most of which abandon clothing-related appearance features. RF-Reid is an end-to-end model for person re-identification that takes a radio frequency trajectory as input and extracts features from the trajectory, enabling the model to obtain accurate human contours for identification. In PRCC, researchers used an Angle-Specific Extractor (ASE) to extract fine-grained angle-specific discriminative features by changing the sampling range of the SPT, and contributed a multi-stream network to aggregate multi-granularity features. BC-Net is a two-branch network that uses clothing templates to retrieve candidate pedestrian images, effectively fusing biometric and clothing features. CASE-Net utilizes a cross-appearance adversarial learning strategy to extract body shape features and structures body shape through image generation of pose variations. FSAM proposes a two-stream framework that learns fine-grained body shape-related information from the body shape stream and transfers it to the appearance stream to supplement clothing-independent feature information in the appearance features.

[0007] Early research on person re-identification models primarily focused on manually designed features and learning better similarity metrics. However, with the development of deep learning, the automatic extraction of better features and the learning of better similarity habit metrics through neural networks have significantly improved the performance of person re-identification. However, these studies simplified the application scenarios, assuming that retrieval only occurs within a short period and that pedestrians' clothing remains unchanged. Therefore, most person re-identification models focus on the color and texture features of pedestrian clothing, which limits the application scenarios. Due to limitations in manpower and time, existing cross-appearance person re-identification datasets have the following drawbacks: 1. The current cross-appearance datasets are relatively small, insufficient to support the training of deep learning models. 2. The background and other biases of the datasets differ significantly from the real world, leading to domain adaptation problems in the models. 3. The short time intervals between source video captures mean that there are no obvious changes in appearance, making it difficult to observe appearance differences. In real-world scenarios, people change their clothing based on factors such as time and weather, making the assumption that pedestrians' appearance remains consistent in the short term no longer applicable. The new scenarios also present new challenges to pedestrian re-identification models, requiring them to no longer rely on clothing-related appearance features, but to focus more on robust feature information such as hairstyle and body shape. Summary of the Invention

[0008] This invention addresses the technical problems of poor performance in long-term cross-appearance pedestrian re-identification, large differences in clothing, and lack of relevant datasets in existing technologies. It provides a generative model-based cross-appearance pedestrian re-identification and detection method. The generative model enhances the appearance dimension of pedestrian images and introduces the generated pedestrian images into the training phase of the model. At the same time, multimodal pedestrian images are introduced. In addition to traditional RGB images, two additional types of images with less correlation to appearance are added: pedestrian outline sketches and pedestrian analytical images. The three modalities use the same backbone network to extract features and are fused for inference.

[0009] The technical solution adopted in this invention is: a cross-appearance pedestrian re-identification and detection method based on a generative model, comprising the following steps:

[0010] Step 1: Input pedestrian images from the pedestrian dataset into the generation model. Different encoders of the generation model obtain the body shape features and appearance features of the pedestrian images respectively. The body shape features and appearance features are exchanged between different pedestrian images. New pedestrian images are generated by the generator. The label information of the new pedestrian images is identified by the discriminator. The new pedestrian images and their label information are saved to the pedestrian dataset.

[0011] Step 2: Take the RGB images of all pedestrian images in the pedestrian dataset as input, and extract the contour information and semantic segmentation information of the pedestrians through a pre-trained edge detection network and a pre-trained pedestrian semantic segmentation network, respectively, to obtain pedestrian contour sketches and pedestrian parsing maps, and save them to the pedestrian dataset;

[0012] Step 3: Input the pedestrian contour sketches, RGB images, and pedestrian resolution images from the pedestrian dataset into the same backbone network to extract features, obtaining f. Contour f RGB and f Parsing Feature fusion yields f Conb Then, inference training is performed, and the trained backbone network performs cross-appearance pedestrian re-identification and detection.

[0013] Furthermore, in step 1, after the pedestrian image is input into the generation model, appearance features and body shape features are obtained by the appearance encoder and body shape encoder, respectively. The appearance features and body shape features of the same pedestrian image are input into the generator to obtain a self-reconstructed pedestrian image. The discriminator identifies the label information of the self-reconstructed pedestrian image. The self-reconstructed pedestrian image is then processed again by the appearance encoder and body shape encoder to obtain appearance features and body shape features, respectively. The appearance features of the self-reconstructed pedestrian image and the body shape features of another self-reconstructed pedestrian image are input into the generator to obtain a new pedestrian image. The discriminator identifies the label information of the new pedestrian image. The new pedestrian image and its label information are saved to the pedestrian dataset.

[0014] Furthermore, in step 1, the appearance encoder is a feature extraction network of the pedestrian re-identification network; the body shape encoder is a network that incorporates a pyramid structure that focuses on different scales.

[0015] Furthermore, during the training of the generative model, the appearance encoder uses an identity loss function for constraint, and the body shape encoder uses an identity label loss function to guide the training of the model.

[0016] Furthermore, in step 2, the edge detection network is pre-trained using the edge detection dataset; the pedestrian semantic segmentation network is pre-trained using the LIP dataset.

[0017] Furthermore, in step 3, the f Contour f RGB and f Parsing The fusion method is weighted splicing.

[0018] Furthermore, in step 3, the backbone network adopts Densenet121.

[0019] Furthermore, in step 3, the loss of the backbone network includes the loss of f. Contourf RGB and f Parsing TriHard loss and the f Conb Loss of identity Appearance loss The formula is as follows:

[0020]

[0021] In equation (1) above, P and K represent the number of pedestrians and the corresponding number of images in a batch, respectively. Given the interval parameter α and the distance metric D, for the k-th image of the p-th person in the batch... Received For the k′-th image of the p-th person in the batch Received For the k′-th image of the p′-th person in the batch Received Constrain its features against the most difficult positive sample Features of the most difficult negative samples The distance; in equations (2) and (3) above, given the image x i and identity tag y i and appearance label z i ,p(y i |x i ) and p(z i |x i ) represent the model recognizing the image as an identity label y. i and appearance label z i The probability distribution.

[0022] Compared with the prior art, the beneficial effects of this invention are:

[0023] 1. This invention introduces a generative model. On one hand, it can enhance data by increasing the diversity of pedestrian appearance in a limited pedestrian dataset, thereby improving the quality of the dataset. On the other hand, the generative model separates appearance-related features and cross-appearance robust features through its appearance encoder and body shape encoder. The appearance-robust features extracted by the body shape encoder are then fused with other features to improve the model's adaptability to appearance changes, resulting in a better-performing pedestrian re-identification model. The images generated by the generative model can fill gaps in the distribution curve of the original dataset. The generated images are more similar to the original dataset in terms of lighting and contrast, closely resembling the actual test set. Compared to combining multiple datasets, the generated images are closer to the test set and are therefore more likely to improve the model's results.

[0024] 2. Because traditional pedestrian re-identification models rely excessively on appearance features, this invention employs pedestrian images from three different modalities. This retains the model's adaptability to traditional pedestrian re-identification tasks while reducing its dependence on appearance. The features used for inference are more adaptable to cross-appearance scenarios. This invention integrates multimodal information into the pedestrian re-identification model, improving cross-appearance target retrieval by considering contour features and human body parsing features—two appearance-independent cues. This guides the model to learn key features of the same pedestrian across different appearances, resulting in more robust performance in cross-appearance scenarios. Attached Figure Description

[0025] Figure 1 This is a flowchart of an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of the generative model according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of the image extraction and pedestrian re-identification model according to an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Embodiments of the present invention provide a cross-appearance person re-identification detection method based on a generative model, such as... Figure 1 As shown, it includes the following steps:

[0030] Step 1: The number of images in the pedestrian dataset has a significant impact on the performance of the deep learning model. Based on the settings and scale of commonly used pedestrian datasets, the number of pedestrian appearances is compared with pose or angle. The current two appearances are insufficient to support model optimization. A generative model is used to combine pedestrians with various appearances to generate other clothing images of pedestrians, forming a large number of appearances for a single pedestrian, thereby guiding the model to learn robust pedestrian features.

[0031] The features of pedestrian images across different appearances can be broadly categorized into two types: one is the pedestrian's own characteristics, such as height and body shape, which are independent of clothing and other appearance features and are the desired features of the model; the other is appearance information such as clothing color and texture, which accounts for a large proportion in traditional pedestrian re-identification. In this embodiment, pedestrian images from the pedestrian dataset are input into the generation model. Different encoders in the generation model are used to obtain the body shape and appearance features of the pedestrian images. Body shape and appearance features are exchanged between different pedestrian images, achieving the exchange of images of different pedestrians and different appearances of the same pedestrian. Then, a new pedestrian image is generated by the generator, and a discriminator is used to identify the label information of the new pedestrian image. This process achieves the purpose of separating the two types of features. Finally, the new pedestrian image and its label information are saved to the pedestrian dataset. Generating new pedestrian images enriches the diversity of pedestrian images, and adding the generated pedestrian images to the model's training phase can improve the quality of the pedestrian dataset and enhance the model's ability to handle cross-appearance problems. Figure 2 As shown, the specific process of data augmentation of pedestrian datasets using generative models is as follows: After pedestrian images from the pedestrian dataset are input into the generative model, appearance features and body features are obtained through appearance encoder and body shape encoder, respectively. The appearance features and body shape features of the same pedestrian image are input into the generator to obtain a self-reconstructed pedestrian image. The discriminator identifies the label information of the self-reconstructed pedestrian image. The self-reconstructed pedestrian image is then processed again by appearance encoder and body shape encoder to obtain appearance features and body shape features, respectively. The appearance features of the self-reconstructed pedestrian image and the body shape features of another self-reconstructed pedestrian image are input into the generator to obtain a new pedestrian image. The discriminator identifies the label information of the new pedestrian image. The new pedestrian image and its label information are saved to the pedestrian dataset.

[0032] The generative model's body encoder and appearance encoder are constrained by different loss functions, ensuring that each encoder focuses on features of different parts of the pedestrian image. Based on the knowledge that traditional pedestrian re-identification algorithms focus on appearance information such as clothing, the appearance encoder uses a feature extraction network commonly used in pedestrian re-identification networks, and therefore uses a common identity loss function for constraint. The body encoder uses a network that incorporates a pyramid structure that focuses on different scales. In traditional pedestrian datasets, pedestrian images do not contain appearance variations, so identity labels and appearance labels are consistent; in this embodiment, pedestrian identity labels are introduced during the training of the generative model, and an identity label loss function is used to guide the model's training. The discriminator of the generative model adversarially improves the quality of the generated images by judging whether they are genuine or fake.

[0033] The quality of training data depends not on its quantity, but rather on its representativeness, so that the model can learn more robust features. The pedestrian images generated by the generative model originate from the training set. Unlike the training set, these images fill gaps in the data distribution curve, enriching the appearance dimension of the pedestrian images. In selecting training images for the generative model, considering the characteristics of long-term appearance differences, the input to the generative model is adjusted according to the time span, selecting pedestrian images with significant appearance differences for training and generation to achieve substantial appearance variations.

[0034] Step 2: The pedestrian outline mainly consists of the pedestrian's edge information, including the important information of the pedestrian's body shape; while the human body map mainly obtains information about the local area of ​​the pedestrian, often used as auxiliary information for pedestrian alignment. Both pedestrian outline information and human body map are robust to appearance changes to a certain extent, and can provide the network with pedestrian information that is robust to appearance changes.

[0035] The process of extracting pedestrian contours and human body resolution images is the prior knowledge extraction stage of the model. Before model training, the RGB images of all pedestrian images in the pedestrian dataset are used as input. These images are processed by an edge detection network pre-trained on an edge detection dataset and a pedestrian semantic segmentation network pre-trained on the LIP dataset to extract pedestrian contour information and semantic segmentation information, respectively, resulting in pedestrian contour sketches and pedestrian resolution images. All images are then saved to the pedestrian dataset along their original paths, such as... Figure 3 As shown.

[0036] Step 3: As Figure 3 As shown, pedestrian contour sketches, RGB images, and analytical pedestrian images from the pedestrian dataset are input into the same backbone network to extract features, resulting in f. Contour f RGB and f Parsing Feature fusion yields f Conb Then, inference training is performed, and the trained backbone network performs cross-appearance pedestrian re-identification and detection.

[0037] Pedestrian contour information and human body analysis images cannot be fully used as features for pedestrian re-identification. Therefore, in addition to introducing the two modalities into the pedestrian re-identification network, the RGB image of the pedestrian is still kept as input. The global features of the RGB pedestrian image and the features of the other two modalities are obtained through the backbone network. These features are mutually supervised with the features of the RGB image and complement each other with more details in the RGB image. Finally, the three features are weighted and concatenated, which effectively reduces the dependence on appearance-related pedestrian features in pedestrian recognition and improves the accuracy of the pedestrian re-identification model.

[0038] The process of pedestrian re-identification generally consists of two stages: feature extraction and distance measurement. The quality of the feature extraction network directly affects the quality of the pedestrian re-identification model. Existing works often use deep networks as the backbone, but due to differences in network structure and the number of network layers, the feature extraction performance varies considerably. This embodiment selects DenseNet121 as the backbone network and conducts experiments with different network depths. While ensuring pedestrian recognition speed, the network size is gradually increased to obtain better pedestrian re-identification performance. Empirically, in the structure of deep neural networks, as the number of network layers increases, the model generally achieves better results, but gradually the gradient vanishing phenomenon occurs, leading to a significant decrease in the model's training effect. DenseNet121 connects the input of each layer with the output of all previous layers, alleviating the gradient vanishing problem while connecting shallow and deep pedestrian features, achieving a combination of global features and fine-grained local features. This approach has good adaptability to pedestrian re-identification, which requires multi-scale features, as shown in Table 1.

[0039] Table 1. Network Structure and Output Dimensions of Densenet121

[0040]

[0041] Existing pedestrian re-identification models often employ the TriHard loss function, selecting the positive sample with the lowest similarity to the anchor image and the negative sample with the highest similarity in a batch to calculate a ternary loss function. This embodiment, to guide the DenseNet121 network in jointly learning these three different modalities of data and their features, introduces identity loss, appearance loss, and TriHard loss, i.e., f... Contour f RGB and f Parsing TriHard loss, and f Conb The backbone network uses five loss functions: identity loss and appearance loss. During parameter tuning, the appearance loss is multiplied by a coefficient λ (0 < λ < 1). The identity loss guides the model to learn appearance-related features, and the combination of identity and appearance loss guides the model to learn pedestrian identity-related features. The TriHard loss makes the model more robust in classifying difficult samples. The TriHard loss calculates the similarity of feature vectors in Euclidean space, while the identity and appearance losses calculate vector similarity in cosine space. The formulas for the three loss functions are as follows:

[0042]

[0043] In equation (1) above, P and K represent the number of pedestrians and the corresponding number of images in a batch, respectively. Given the interval parameter α and the distance metric D, for the k-th image of the p-th person in the batch... Received For the k′-th image of the p-th person in the batch Received For the k′-th image of the p′-th person in the batch Received Constrain its features against the most difficult positive sample Features of the most difficult negative samples The distance; in equations (2) and (3) above, given the image x i and identity tag y i and appearance label z i ,p(y i |x i ) and p(z i |x i ) represent the model recognizing the image as an identity label y. i and appearance label z i The probability distribution.

[0044] To evaluate the performance of the generative model-based cross-appearance person re-identification model, this embodiment compares it (Ours) with other state-of-the-art (SOTA) person re-identification methods on the NKUP dataset. The experimental results are shown in Table 2. This embodiment uses Rank 1 and mAP to evaluate the model's performance; the former represents the accuracy of the first-ranked person image in the query list, and the latter is the average accuracy of all query samples. The baseline model in Table 2 is a network model without generative model-based data augmentation. The experimental results show that the proposed method has significant advantages, especially in cross-appearance person re-identification scenarios, achieving the best results and verifying the model's improvement in cross-appearance person re-identification.

[0045] Table 2 Comparison of Experimental Results

[0046]

[0047] The present invention has been described in detail above through embodiments, but the content described is only an exemplary embodiment of the present invention and should not be considered as limiting the scope of the present invention. The scope of protection of the present invention is defined by the claims. Any technical solutions designed by those skilled in the art using the technical solutions described in the present invention, or designed by those skilled in the art under the inspiration of the technical solutions of the present invention, within the substance and protection scope of the present invention, to achieve the above-mentioned technical effects, or any equivalent changes and improvements made to the scope of the application, should still fall within the patent protection scope of the present invention.

Claims

1. A method for cross-appearance pedestrian re-identification and detection based on a generative model, characterized in that, Includes the following steps: Step 1: Input pedestrian images from the pedestrian dataset into the generation model. Different encoders of the generation model obtain the body shape features and appearance features of the pedestrian images respectively. The body shape features and appearance features are exchanged between different pedestrian images. New pedestrian images are generated by the generator. The label information of the new pedestrian images is identified by the discriminator. The new pedestrian images and their label information are saved to the pedestrian dataset. Step 2: Take the RGB images of all pedestrian images in the pedestrian dataset as input, and extract the contour information and semantic segmentation information of the pedestrians through a pre-trained edge detection network and a pre-trained pedestrian semantic segmentation network, respectively, to obtain pedestrian contour sketches and pedestrian parsing maps, and save them to the pedestrian dataset; Step 3: Input the pedestrian contour sketches, RGB images, and analytical pedestrian images from the pedestrian dataset into the same backbone network to extract features, and obtain... , and Feature fusion yields Then, inference training is performed, and the trained backbone network performs cross-appearance pedestrian re-identification and detection. In step 1, after a pedestrian image is input into the generation model, appearance features and body shape features are obtained through an appearance encoder and a body shape encoder, respectively. The appearance features and body shape features of the same pedestrian image are input into the generator to obtain a self-reconstructed pedestrian image. The discriminator identifies the label information of the self-reconstructed pedestrian image. The self-reconstructed pedestrian image is then processed again by the appearance encoder and body shape encoder to obtain appearance features and body shape features, respectively. The appearance features of the self-reconstructed pedestrian image and the body shape features of another self-reconstructed pedestrian image are input into the generator to obtain a new pedestrian image. The discriminator identifies the label information of the new pedestrian image. The new pedestrian image and its label information are saved to the pedestrian dataset. In step 1, the appearance encoder is the feature extraction network of the pedestrian re-identification network; the body shape encoder is a network that incorporates a pyramid structure that focuses on different scales. When the generative model is trained, the appearance encoder uses an identity loss function for constraint, and the body shape encoder uses an identity label loss function to guide the training of the model.

2. The cross-appearance pedestrian re-identification and detection method based on a generative model as described in claim 1, characterized in that, In step 2, the edge detection network is pre-trained using the edge detection dataset; the pedestrian semantic segmentation network is pre-trained using the LIP dataset.

3. The cross-appearance pedestrian re-identification and detection method based on a generative model as described in claim 1, characterized in that, In step 3, the , and The fusion method is weighted splicing.

4. The cross-appearance pedestrian re-identification and detection method based on a generative model as described in claim 1, characterized in that, In step 3, the backbone network adopts Densenet121.

Images