A pedestrian re-identification method based on deepfake clothing

Abstract

This invention discloses a pedestrian re-identification method based on deepfake appearance, including a training phase and a testing phase. The overall framework consists of two branches: an original image feature extraction branch network E and a deepfake appearance feature extraction branch network M. First, an existing deepfake appearance model is used to perform "appearance transformation" on the images in the training dataset, and the transformed images are saved to the training dataset. During the training phase, both branches participate in the training. Taking the deepfake appearance feature extraction branch M as an example, the deeply transformed pedestrian images are input into the backbone network to extract features, which are then separated into identity features and clothing features through an attention mechanism. The identity features extracted by these two branches are brought as close together as possible to extract more robust identity features. In the testing phase, only network E is used to extract identity features for the input images for identity inference. This invention can complete the pedestrian re-identification task and effectively reduces the negative impact of appearance changes such as pedestrian appearance transformation on pedestrian re-identification.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and pattern recognition, and particularly relates to a pedestrian re-identification method based on deep clothing transformation. Background Technology

[0002] In recent years, with the widespread application of surveillance equipment, pedestrian identification technologies have received increasing attention. Pedestrian identification focuses on using captured pedestrian images to search for pedestrians with the same identity in a pedestrian database to confirm the identity of the captured pedestrian. Pedestrian identification has broad application scenarios in the Internet of Things and big data environments, including smart cities and smart security. Currently, major pedestrian re-identification technologies are closely related to pedestrian identification, and pedestrian re-identification has recently gained widespread attention, achieving significant performance improvements on public datasets. However, the high cost of pedestrian identification annotation in real-world scenarios, and the significant differences in lighting, background, and pose of pedestrian images obtained in different domains (scenes), pose significant challenges to the application of pedestrian re-identification in real-world scenarios. Current mainstream deep learning methods generally focus on inferring pedestrian appearance information, which is difficult to apply in real-world scenarios where pedestrians frequently change their appearance.

[0003] Most current algorithms use attention mechanisms to focus the model on more distinctive regions to improve performance. However, in real-world scenarios, pedestrians frequently change clothes, and the same pedestrian can have different appearance features when wearing different outfits. Focusing only on local areas will result in poor generalization performance of the model. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing pedestrian re-identification algorithms in scenarios involving clothing changes, and to provide a pedestrian re-identification method based on deep clothing changes. This invention can complete the pedestrian re-identification task and effectively reduce the negative impact of appearance changes such as clothing changes on pedestrian re-identification.

[0005] The objective of this invention is achieved through the following technical solution: a pedestrian re-identification method based on deep clothing transformation, comprising the following steps:

[0006] 1) Use a deep clothing model and pre-selected clothing templates to dress up pedestrians in the training set images, save and add them to the training set.

[0007] 2) During the training phase, the original image feature extraction branch network E and the deep clothing transformation feature extraction branch network M are used to extract the identity features and clothing features of the original image and the deep clothing transformation image, respectively. Networks E and M are then trained to improve the classification performance of the extracted features.

[0008] 3) During the training phase, networks E and M are trained so that the identity features extracted by E and M are closer together.

[0009] 4) During the testing phase, only the original image feature extraction branch network E is used to extract identity information. The identity information is used to measure similarity and infer identity. The one with the highest similarity is the final matching result.

[0010] Further, in step 2), the original image feature extraction branch network E and the deep clothing transformation feature extraction branch network M extract the identity features and clothing features of the original image and the deep clothing transformation image, respectively. The specific process is as follows: the image is first input into the backbone network to extract features f. s Then, through an attention mechanism, it is separated into identity features and clothing features:

[0011] f s =Backbone (I)

[0012] f clo =Atten(f s )*f s

[0013] f ID =(1-Atten(f) s ))*f s

[0014] Among them, f clo For clothing information features, f ID For identity information features, f s The main network extracts features from pedestrian images as input, where I is the image input, and Atten(f) is the input image. s ) for the application of attention mechanisms to f s The resulting attention map.

[0015] Furthermore, in step 2), the clothing features and identity features separated by the attention mechanism are used for supervised training using a classification loss:

[0016]

[0017]

[0018] Here, CE represents a classification loss.

[0019] Further, in step 3), networks E and M extract the identity features and clothing features of the original image and the deepfake image, respectively. E and M are trained to minimize the distance between the identity features of the two branches:

[0020]

[0021] Where Ic represents the image after deep customization, and I represents the original image.

[0022] Further, step 4) specifically involves: during the testing phase, without using the deep costume feature extraction branch network M, a pedestrian image is input into the original image feature extraction branch network E to extract identity features for pedestrian identity inference.

[0023] Furthermore, in step 1), the deep costume change model adopts PF-AFN, etc.

[0024] Furthermore, in step 2), the backbone networks of networks E and M adopt the ResNet-50 network structure.

[0025] Furthermore, in step 2), the attention mechanism consists of channel attention and spatial attention.

[0026] Furthermore, in step 2), the classification loss adopts a classification loss based on cross-entropy and a triplet loss.

[0027] Further, in step 3), the MSE metric function is used to measure the distance between the identity features extracted by networks E and M.

[0028] The beneficial effects of this invention are as follows: This invention separates identity features and clothing features through attention, extracting more distinctive identity features, and then using these features for inference, improving the model's adaptability to pedestrians changing clothes. Simultaneously, by obtaining images of the same pedestrian with the same body shape and posture wearing different clothing through deep clothing swapping, it is more conducive to the model learning identity features unrelated to clothing. In real-world scenarios, pedestrians frequently change clothes. Conventional deep learning methods focus on appearance features for inference, and images of the same pedestrian wearing different clothes may lead to misjudgments due to significant appearance differences. This invention is expected to reduce the negative impact of pedestrian clothing swapping on pedestrian re-identification in real-world scenarios to a certain extent, thereby improving recognition accuracy in real-world situations. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the overall structure of the pedestrian re-identification network of the present invention;

[0030] Figure 2 This is a flowchart of the training phase of the present invention;

[0031] Figure 3 This is a flowchart of the testing phase of the present invention;

[0032] Figure 4 This is a schematic diagram illustrating an example of the attention mechanism used in this invention;

[0033] Figure 5 This is a schematic diagram of the matching results sorted by similarity according to an embodiment of the present invention. Detailed Implementation

[0034] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0035] like Figure 1 As shown, this invention discloses a pedestrian re-identification method based on deepfake technology, comprising a training phase and a testing phase. The overall framework is divided into two branches: an original image feature extraction branch network E and a deepfake feature extraction branch network M. Before training begins, offline data augmentation is used to augment the images in the training dataset using an existing deepfake model, and the augmented images are then added to the training dataset. During the training phase, both branches participate in the training. The original image feature extraction branch network E and the deepfake feature extraction branch network M extract the identity features and clothing features of the original image and the deeply augmented image, respectively. Networks E and M are trained to improve the classification performance of the extracted features, and to make the identity features extracted by networks E and M closer together. Taking the deepfake feature extraction branch M as an example, the deeply augmented pedestrian image is input into the backbone network to extract features, which are then separated into identity features and clothing features through an attention mechanism. The identity features extracted by these two branches are brought as close together as possible to extract more robust identity features. After training, during the testing phase, for the input image, the deep costume feature extraction branch network M is not used; only the original image feature extraction branch network E is used to extract identity features for identity inference. Specifically, this includes the following steps:

[0036] 1) Using a deep clothing swapping model and pre-selected clothing templates, the pedestrians in the training set images are dressed up, saved, and added to the training set. The deep clothing swapping model used can be any currently available model, and the clothing templates can be any clothing, with no style or feature requirements.

[0037] 2) such as Figure 2 As shown, during the training phase, the original image feature extraction branch network E and the deep clothing transformation feature extraction branch network M are used to extract the identity features and clothing features of the original image and the deep clothing transformation image, respectively. Networks E and M are trained to improve the classification performance of the extracted features. Specifically, pedestrian images are input into the backbone networks of networks E and M to extract features, which are then separated into identity features and clothing features through an attention mechanism.

[0038] f s =Backbone(I)

[0039] f clo =Atten(f s )*f s

[0040] f ID =(1-Atten(f) s ))*f s

[0041] Where I is the image input, f s Input the features extracted by the backbone network for the pedestrian image I; Atten(f s ) for the application of attention mechanisms to f s The resulting attention map; f clo For clothing information features, f ID For identity information features. The backbone networks of networks E and M can be any current backbone network structure, such as ResNet, VGGNet, etc.; the attention mechanism can be any current attention module.

[0042] Clothing and identity features separated by the attention mechanism are then used for supervised training using a classification loss method.

[0043]

[0044]

[0045] Where E represents the original image feature extraction branch network, M represents the deep costume feature extraction branch network, and CE represents a classification loss, which can be any loss used for classification.

[0046] 3) such as Figure 2 As shown, during the training phase, networks E and M are trained to make the identity features extracted by E and M closer together:

[0047]

[0048] Where Ic represents the image after deep customization, and I represents the original image.

[0049] 4) such as Figure 3 As shown, in the actual testing phase, only the original image feature extraction branch network E was used to extract identity information, and this identity information was used for similarity measurement and identity inference. Specifically, an image is input into network E to extract identity features, and these identity features are used for similarity measurement and identity inference.

[0050] The implementation process of one embodiment of the present invention is as follows:

[0051] 1) Use a deep clothing model and pre-selected clothing templates to dress up pedestrians in the training set images, save and add them to the training set; use the third-party deep clothing model PF-AFN (CVPR2021) to dress up pedestrians in the training set images.

[0052] 2) During the training phase, the original image feature extraction branch network E and the deep clothing transformation feature extraction branch network M are used to extract the identity features and clothing features of the original image and the deep clothing transformation image, respectively. Networks E and M are trained to make the extracted features have better classification performance.

[0053] The backbone networks of networks E and M adopt the ResNet-50 network architecture, such as... Figure 4 As shown, the attention mechanism consists of channel attention and spatial attention, and the final attention Atten is derived from the channel attention graph A. cha Spatial attention map A spa Multiplying them together gives:

[0054] A cha =sigmoid(Relu(Conv(Relu(Conv(GAP(f s ))))))

[0055] A spa =softmax(Relu(Conv(CGAP(f)) s ))))

[0056] Atten = A spa *A cha

[0057] Among them, GAP, CGAP, Conv, ReLU, and sigmoid are global average pooling, channel-oriented global average pooling, convolutional layer, ReLU activation layer, and sigmoid activation layer, respectively.

[0058] The classification loss is based on cross-entropy. CE and triplet loss L TL The original images use 11 pre-labeled clothing categories, with the categories labeled based on the color and style of the clothing. The clothing category labels for the in-depth transformation images are labeled according to the clothing template used.

[0059]

[0060]

[0061] CE=L CE +L TL

[0062]

[0063]

[0064] Here, CE represents a classification loss. i This represents the true label of sample i. f represents the predicted label of sample i. N represents the number of samples. a f represents the identity or clothing features extracted from an anchor sample via network E or network M. p f represents the feature corresponding to a positive sample that belongs to the same identity as the anchor sample. n The feature represents the negative sample that belongs to a different identity from the anchor sample, and α represents the marginal value by which the positive and negative samples are expected to be pushed further apart.

[0065] 3) During the training phase, networks E and M are trained to make the identity features extracted by E and M closer together. The Mean Square Error (MSE) metric is used to measure the distance between the identity features extracted by networks E and M. By training networks E and M, the distance between the identity features extracted by E and M is made closer.

[0066]

[0067] Where Ic represents the image after deep customization, and I represents the original image.

[0068] 4) In the actual testing phase, the deep costume-swapping feature extraction branch network M is not retained; only the original image feature extraction branch network E is used to extract identity information. This identity information is then used for similarity measurement and identity inference to improve the method's robustness to costume-swapping. The final results are sorted by similarity. An example challenge image matching result is shown below. Figure 5 As shown, the result with the highest similarity is the final matching result.

[0069] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A pedestrian re-identification method based on deepfake clothing, characterized in that, Includes the following steps: 1) Use a deep clothing model and pre-selected clothing templates to dress up pedestrians in the training set images, save and add them to the training set; 2) During the training phase, the original image feature extraction branch network E and the deep clothing transformation feature extraction branch network M are used to extract the identity features and clothing features of the original image and the deep clothing transformation image, respectively, and networks E and M are trained to make the extracted features have better classification performance. Among them, the original image feature extraction branch network E and the deep clothing transformation feature extraction branch network M extract the identity features and clothing features of the original image and the deep clothing transformation image, respectively. The specific process is as follows: the image is first input into the backbone network to extract features f. s Then, through an attention mechanism, it is separated into identity features and clothing features: f s =Backbone(I) f clo =Atten(f s )*f s f ID =(1-Atten(f s ))*f s Among them, f clo For clothing information features, f ID For identity information features, f s The pedestrian image is input as the feature extracted by the backbone network, where I is the image input, and Atten(f) is the feature extracted by the backbone network. s ) for the application of attention mechanisms to f s The resulting attention map; Clothing and identity features separated by the attention mechanism are then used for supervised training using a classification loss method. Here, CE represents a classification loss; 3) During the training phase, networks E and M are trained so that the identity features extracted by E and M are closer together; 4) During the testing phase, only the original image feature extraction branch network E is used to extract identity information. The identity information is used to measure similarity and infer identity. The one with the highest similarity is the final matching result.

2. The pedestrian re-identification method based on deep clothing transformation according to claim 1, characterized in that, In step 3), networks E and M extract the identity features and clothing features of the original image and the deepfake image, respectively; E and M are trained to minimize the distance between the identity features of the two branches: Where Ic represents the image after deep customization, and I represents the original image.

3. The pedestrian re-identification method based on deep clothing transformation according to claim 1, characterized in that, Step 4) Specifically, during the testing phase, without using the deep costume feature extraction branch network M, a pedestrian image is input into the original image feature extraction branch network E to extract identity features for pedestrian identity inference.

4. The pedestrian re-identification method based on deep clothing transformation according to claim 1, characterized in that, In step 1), the deep costume change model uses PF-AFN.

5. The pedestrian re-identification method based on deep clothing transformation according to claim 1, characterized in that, In step 2), the backbone networks of networks E and M adopt the ResNet-50 network structure.

6. The pedestrian re-identification method based on deep clothing transformation according to claim 1, characterized in that, In step 2), the attention mechanism consists of channel attention and spatial attention.

7. The pedestrian re-identification method based on deep clothing transformation according to claim 1, characterized in that, In step 2), the classification loss uses classification loss based on cross-entropy and triplet loss.

8. The pedestrian re-identification method based on deep clothing transformation according to claim 2, characterized in that, In step 3), the MSE metric function is used to measure the distance between the identity features extracted by networks E and M.

Images