A method for pedestrian re-identification after changing clothes based on deep mutual learning

Abstract

The present invention proposes a method for pedestrian re-identification after changing clothes based on a deep mutual learning strategy, which addresses the problem of low recognition accuracy of the pedestrian re-identification dataset PRCC after changing clothes. First, a human body parsing network is used to extract a human body mask from the original image, thereby obtaining appearance features that are robust to clothing changes. Then, the original image and the image with the human body mask extracted are input into two backbone networks respectively, and the two backbone networks learn to extract the appearance features and shape features of the pedestrian respectively. Afterwards, the output distribution difference of the two networks is reduced by MMD loss and KL divergence loss, so that the appearance branch backbone network used for prediction can obtain the ability to extract shape features that are robust to clothing changes. At the same time, the MMD loss is used to reconstruct the vector so that the backbone network can obtain the ability to extract shape features while maintaining the original ability to extract appearance features. In the prediction stage, only the output of the appearance branch backbone network is used to retrieve pedestrians, which can shorten the retrieval time.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and pattern recognition, and relates to a method for re-identifying pedestrians after changing clothes based on a deep mutual learning strategy. It is an innovative application technology of deep mutual learning. Background Art

[0002] Person re-identification (Pedestrian Re-ID) uses computer vision techniques to determine whether a pedestrian in an image is a specific person. Due to varying lighting conditions, postures, and image resolutions, retrieving a specific person from images captured by different cameras is challenging. Person re-ID can be widely applied in fields such as intelligent security and intelligent transportation. Existing Pedestrian Re-ID methods mostly assume that a pedestrian's clothing remains largely unchanged. However, in real-world scenarios, such as where a suspect might change clothing to evade detection, traditional Pedestrian Re-ID methods are ineffective.

[0003] Traditional person re-identification methods primarily include metric learning and deep learning. Metric learning maps the features of each person into a more discriminative metric space, then searches for the person within this space. Deep learning is currently the mainstream approach for person re-identification. A wide variety of deep learning methods for person re-identification have emerged, demonstrating impressive results on large datasets. However, as mentioned above, these methods assume that a person's clothing remains unchanged over a short period of time. For practical application of person re-identification technology, the issue of clothing changes in person re-identification urgently needs to be addressed.

[0004] The proposed method for re-identifying people with clothing changes is a classification model based on mutual learning. Through a two-stream network, this method improves the prediction branch's ability to extract features that are robust to changes in clothing, such as body shape. Given the limited capabilities of existing re-identification methods for re-identifying people with clothing changes and the specific characteristics of the recognition task, this method better meets the requirements of re-identifying people with clothing changes. Summary of the Invention

[0005] To address the difficulty of existing pedestrian re-identification methods in handling the issue of changing clothes, this paper proposes a method that can handle the problem of pedestrian re-identification after changing clothes by combining the concept of mutual learning. This method can achieve fast and efficient recognition of pedestrians who have changed clothes.

[0006] A method for re-identifying pedestrians after changing clothes based on the idea of mutual learning, characterized by comprising the following steps:

[0007] Step A: First, use the existing human body parsing network to parse out the pedestrians in the input image, that is, generate a pedestrian mask. This mask is independent of the color of the pedestrian's clothing and contains the pedestrian's appearance information. This appearance information is more robust to changes in pedestrian clothing.

[0008] Step B: The original image and the masked image are fed into two identical but parameter-unshared backbone networks (OSNet) to extract appearance features and shape features, respectively. The two branches correspond to two classifiers. In addition to using the outputs of the two classifiers to calculate the cross-entropy classification loss in the training phase, the MMD loss is also used between the two to enable the appearance backbone network to acquire some ability to extract shape knowledge.

[0009] Step C: The outputs of the two backbone networks are input into two small convolutional networks respectively. The KL divergence is used to calculate the distribution difference of the outputs of the two small convolutional networks and used as the loss to improve the feature extraction ability of the appearance backbone network that is robust to clothing changes, that is, the appearance features.

[0010] Step D: In order to reduce the negative impact of mutual learning on the original feature extraction capabilities of the appearance branch and the shape branch, a reconstruction module is added after each network, and the MMD loss is calculated between the reconstructed vector and the vector output by the backbone for prediction.

[0011] 3. Compared with the prior art, the present invention has the following advantages:

[0012] 1) This invention addresses the inadequacy of traditional person re-identification methods in addressing the issue of clothing changes and proposes a mutual learning-based person re-identification method. As shown in the figure, mutual learning is used to reduce the difference in the output distributions of the two networks, allowing knowledge to flow between the surface flow and the shape flow. This allows the surface flow to improve its ability to extract features unrelated to clothing changes, namely shape features. A reconstruction module is also added to prevent the loss of feature extraction capabilities for pedestrians wearing the same clothing during the mutual learning process. This mutual learning-based person re-identification method effectively improves the recognition accuracy of pedestrian re-identification under clothing changes.

[0013] 2) This invention applies deep mutual learning to re-identification of people in costume changes, representing a cutting-edge exploratory application of artificial intelligence in pattern recognition. This method improves the accuracy of re-identification of people in costume changes, which is of great significance for promoting the development of artificial intelligence methods and their application in various fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Original input image

[0015] Figure 2Input image mask after extracting the mask

[0016] Figure 3 Deep Mutual Learning Framework DETAILED DESCRIPTION

[0017] In one embodiment of the present invention, pedestrian images are collected from the PRCC dataset, which contains 33,698 images of 221 pedestrians. Each pedestrian is captured from three cameras at different angles, and one of the images captures the pedestrian in a different outfit. Of these, 17,896 images are used for training, 5,002 for validation, and 10,800 for testing.

[0018] Note: In the following descriptions, the subscript a indicates that the module or tensor is located in the outer flow, and the subscript s indicates that the module or tensor is located in the outer flow. The subscripts p and q are only used to describe the variables in the formula.

[0019] The present invention provides a method for re-identifying pedestrians in clothing changes based on a deep mutual learning strategy. The specific implementation method includes: 1. Extracting human body masks from input images

[0020] The original image is recorded as Use the existing human body parsing network SCHP to obtain the human body mask, mark the background in the image as black, and the pedestrian as white, and the obtained image is recorded as mask, that is,

[0021]

[0022] Original image Examples include Figure 1 As shown, the image mask example after extracting the mask is as follows Figure 2 shown.

[0023] 2. Learn from each other

[0024] The original image The mask image mask is input into two OSNet backbone networks (both backbone networks use osnet_x1_0 version) respectively. a (·) and the shape flow f s (·), and output the corresponding feature tensors respectively t s =f s (mask), the two feature tensors are passed through two multi-layer perceptron modules (each multi-layer perceptron module consists of a global average pooling layer, a 1000-dimensional to 512-dimensional and a 512-dimensional to category number dimension fully connected layer in sequence) cls a (·) and cls b (·), and the vector used for classification training is obtained, namely

[0025] u a =cls a (t a ),u s =cls s (t s )

[0026] The prediction results corresponding to these two vectors and and the corresponding true values and Do label smoothing cross entropy loss to make the outer flow trunk f a (·) and the outer flow trunk f s (·) It can improve the feature extraction capabilities of appearance features and shape features from the classification learning process respectively.

[0027] The label smoothed cross entropy loss is

[0028]

[0029] in

[0030] N is the batch size

[0031] Then, KL divergence loss and MMD loss are used to reduce the distribution difference of the two backbone outputs, and the outer flow backbone is enabled to obtain the ability to extract shape features through mutual learning. The specific approach is: the feature tensors t of the two backbone outputs are respectively a and t s Input two convolution blocks (each convolution block consists of three convolution kernels, all of which are 1*1, with a step size of 1, and the input and output channels are divided into 512 to 256, 256 to 64, and 64 to 32 convolution layers arranged in sequence and a global average pooling layer) CONV a (·) and CONV s (·) is used to reduce the channel dimension of the tensor to prevent the calculation distortion caused by too high a dimension, and then the KL divergence is calculated, that is,

[0032] m a , v a =CONV a (t a ), m s , v s =CONV s (t s )

[0033] where m a and m s CONV a (·) and CONVs (·) The output of the third convolution layer is a 32-channel tensor, v a and v s t a and t s After CONV a (·) and CONV s The vector output after the global average pooling layer in (·).

[0034] KL divergence formula, where N is the batch size

[0035]

[0036] MMD Maximum Mean Difference Formula

[0037] μ(·) and σ(·) are KL losses calculated for the mean and variance respectively.

[0038] L KL =D KL (m a ||m s )+D KL (m s ||m a )

[0039] Calculating MMD loss

[0040]

[0041] L MMD =D MMD (u a ||u s )

[0042] In order to prevent the reduction of the ability to extract the appearance and shape features of each network in the process of reducing the difference in the output distribution of the two networks, the output m before the two networks are a and m s Input two convolution blocks symmetrical to the previous convolution block, namely reconstruction modules (each reconstruction module consists of three convolution kernels, all of which are 1*1, with a step size of 1, and the input and output channels are divided into 32 to 64, 64 to 256, and 256 to 512 convolution layers arranged in sequence and a global average pooling layer) CONV′ a (·) and CONV′ s (·),Right now

[0043] v a ′=CONV′ a (m a ), v s ′=CONV′ s (ms )

[0044] Among them, v a ′ and v s ′ are respectively CONV′ a (·) and CONV′ s (·) The output of the third convolution layer is a 512-dimensional vector after global average pooling. These two 512-dimensional vectors are respectively compared with v before entering the two convolution blocks. a and v s The MMD loss is calculated separately to reduce the difference between vectors, so that the two backbone networks can retain the ability to extract appearance features and shape features respectively.

[0045] Calculate MMD loss:

[0046] L MMD1 =D MMD (v a ||u′ a )

[0047] L MMD2 =D MMD (v s ||u′ s )

[0048] The final loss is the weighted sum of all losses:

[0049] L=0.3*L cls +0.2*(L MMD +L MMD1 +L MMD2 )+0.2*L KL

[0050] The learning rate during training is 0.001, the optimizer is Adam, the number of training rounds is 100, and the batch size is 32.

[0051] This concludes the training.

[0052] 3. Prediction

[0053] After the above training, the backbone network not only maintains the original pedestrian re-identification feature extraction capability, but also has the ability to extract features that are robust to clothing changes, namely appearance features. At the same time, in order to speed up the prediction stage, during the prediction, the query image and gallery image to be retrieved are input into the appearance backbone respectively, and the query corresponding output vector v is used. ag And gallery corresponding output vector v agThe cosine similarity between the query image and the gallery image is calculated. The pedestrian corresponding to the vector with the highest similarity to the query in the gallery is the person being retrieved, thus completing the costume-changing person re-identification task. The comparison results of the costume-changing person re-identification method based on mutual learning and other existing methods are shown in Table 1.

[0054] Table 1 Comparison of the results of cross-dressing pedestrian re-identification based on mutual learning and other methods

[0055]

Claims

1. A method for pedestrian re-identification after changing clothes based on mutual learning, characterized by The following steps are involved: Step A: First, use the existing human parsing network to parse the pedestrians in the input image, that is, generate a pedestrian mask. This mask is independent of the pedestrian's clothing color and contains the pedestrian's appearance information; Step B: The original image and the masked image are fed into two identical but parameter-unshared backbone networks, OSNet, to extract appearance features and shape features, respectively. The two branches correspond to two classifiers. The outputs of the two classifiers are used to calculate the cross-entropy classification loss during the training phase. MMD loss is also used between the two classifiers to enable the appearance backbone network to acquire some ability to extract shape knowledge. Step C: The outputs of the two backbone networks are fed into two small convolutional networks respectively. The KL divergence is used to calculate the distribution difference between the outputs of the two small convolutional networks and used as the loss to improve the feature extraction ability of the appearance backbone network that is robust to clothing changes, i.e., the appearance features. Step D: Add a reconstruction module after each of the two networks and calculate the MMD loss between the reconstructed vector and the vector output by the backbone for prediction.

2. The method for re-identifying pedestrians after changing clothes based on mutual learning according to claim 1, characterized in that: The step A comprises: The original image is denoted as pic, and the existing human body parsing network SCHP is used to obtain the human body mask. The background in the image is marked as black and the pedestrian is marked as white. The image obtained by SCHP is denoted as mask.

3. The method for re-identifying pedestrians after changing clothes based on mutual learning according to claim 1 is characterized by: The subscript a indicates that this module or tensor is located in the outer flow, and the subscript s indicates that this module or tensor is located in the outer flow; The subscripts p and q are only used to describe the variables in the formula when introducing the formula; The step B comprises: inputting the original image pic and the mask image mask into two OSNet backbone network outer flows f respectively. a (·) and the shape flow f s (·), and output the corresponding feature tensor t respectively. a =f a (pic),t s =f s (mask), the two feature tensors are respectively passed through two pooling fully connected modules cls a (·) and cls b (·), we get a vector with the length of the number of categories used for classification training, namely u a =cls a (t a ),u s =cls s (t s ) The prediction results corresponding to these two vectors and and the corresponding true values and Perform cross entropy loss; The label smoothed cross entropy loss is: in ε = 0.1, N is the batch size, and then MMD loss is used to reduce the distribution difference of the two backbone outputs. The outer flow backbone is able to extract shape features through mutual learning. The formula for the maximum mean difference of MMD is: μ(·) and σ(·) are the mean and variance of the shape flow output u respectively. s And external flow output u a MMD loss between: L MMD =D MMD (u a ||u s )。 4. The method for re-identifying pedestrians after changing clothes based on mutual learning according to claim 1 is characterized by: The step C comprises: The feature tensors t output by the two backbones are respectively a and t s Input two convolution blocks CONV a (·) and CONV s (·) to reduce the channel dimension of the tensor, and then calculate the KL divergence, and use the KL divergence to continue to constrain, that is, m a ,v a =CONV a (t a ),m s ,v s =CONV s (t s ) where m a and m s CONV a (·) and CONV s (·) The output of the third convolution layer is a 32-channel tensor, v a and v s t a and t s After CONV a (·) and CONV s The vector output after the global average pooling layer in (·); Calculate KL loss, where N is the batch size

Images