A cross-domain person re-identification method based on joint training based on hybrid memory

By using a joint training method of hybrid memory and the Multi-Label-Cross-Entropy Loss loss function in cross-domain pedestrian recognition, combining the source domain and target domain data sets, the insufficient utilization and noise of labeled data in the prior art are solved. The impact problem achieves better model generalization performance and practical application effects.

CN113536979BActive Publication Date: 2024-11-12HANGZHOU DIANZI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110720404.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-28
Publication Date
2024-11-12
Estimated Expiration
2041-06-28

AI Technical Summary

Technical Problem

The existing cross-domain pedestrian re-identification method is inefficient when utilizing the source domain marked data and insufficient utilization of unlabeled data, resulting in limited application value in the actual environment, especially when the data difference is large, performance is poor, and pseudo- The noise of label generation has a great impact.

Method used

Using a joint training method based on hybrid memory, the ResNet-50 model is pre-trained on ImageNet, combined with the source domain and the target domain dataset for joint training, and using the Multi-Label-Cross-Entropy Loss (MLCEL) loss function and hybrid memory Module, generate multi-category labels for optimization training, use data augmentation and similarity sorting to form a positive sample set, and update feature representations to improve model performance.

Benefits of technology

Effectively utilize the source domain marked data and the target domain unlabeled data to improve the generalization performance of the model, reduce domain differences, avoid the influence of clustering noise, and achieve widespread application in actual environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113536979B_ABST
    Figure CN113536979B_ABST
Patent Text Reader

Abstract

The present invention discloses a cross-domain pedestrian re-identification method based on joint training of hybrid memory. Firstly, data set preprocessing is performed, and ResNet-50 pre-trained on ImageNet is pre-trained by using source domain data set to obtain a baseline model. Then, the source domain data set and the target domain data set are combined to form a joint data set, and the baseline model is trained and optimized by using the joint data set. By jointly training the source domain and target domain data sets, the method of the present invention can further utilize the rich annotation information in the source domain data set, and can further reduce the domain difference, thereby achieving better adaptive effect. By generating multi-category labels based on similarity and hybrid memory, similar images can be searched in the source domain and the target domain at the same time, so as to further improve the performance of the model and avoid clustering noise similar to that in clustering-based algorithms.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of pedestrian re-identification, and in particular aims at the problem of cross-domain pedestrian re-identification, and specifically relates to a joint training method based on a hybrid memory. Background Art

[0002] As deep learning technologies represented by convolutional neural networks (CNNs) have made significant breakthroughs in the field of computer vision, the field of person re-identification has also begun to adopt research methods based on deep learning, combining feature extraction and distance metrics into an end-to-end framework for joint learning and collaborative optimization. In addition, since the person re-identification model based on deep learning can learn more abstract high-level feature expressions by building a multi-layer model with nonlinear characteristics, it can realize the discovery of implicit information and complex structures of data based on a large amount of data, bringing the performance of person re-identification to a new level. Traditional deep learning-based person re-identification mostly adopts supervised learning, in which the input image has identity annotation information and other auxiliary information. Thanks to the ability of deep neural network models and the birth of a large number of annotated data sets, the former is mainly represented by the introduction of ResNet, and the latter is mainly marked by the publication of Market1501 and DukeMTMC-reID. At present, the mainstream deep learning-based person re-identification can basically reach the theoretical maximum value on a single data set through supervised learning.

[0003] Although supervised person re-identification can achieve very good results on existing datasets, such technology still cannot be used in actual environments due to the labor and time cost of labeling data. Although labeling data is time-consuming and labor-intensive, it is much easier to obtain unlabeled data in actual environments. Based on this setting, cross-domain person re-identification came into being, combining labeled source domain data with unlabeled target domain data to achieve good performance. There are currently two main methods for cross-domain person re-identification:

[0004] 1) The first method attempts to find a common space in which the distribution of the source domain and the target domain can be aligned, and then learn domain-invariant features. This method often performs poorly when the data difference between the source domain and the target domain is large, and has limited application value in complex practical environments;

[0005] 2) The second method is based on the prediction of pseudo-labels. A series of clustering methods are used to generate pseudo-labels for unlabeled data in the target domain, which are then provided to the network for training. However, this method is extremely dependent on the quality of the generated pseudo-labels. Once the pseudo-labels contain too much noise, the performance of the network will drop significantly.

[0006] Another common disadvantage of the above two methods is that they do not make much use of the labeled data in the source domain. They often only use this part of data to train a baseline model and then abandon it. Summary of the invention

[0007] In view of the problems existing in the existing methods, the present invention proposes a cross-domain pedestrian re-identification method based on joint training of hybrid memory, which can effectively utilize the labeled data in the source domain and the unlabeled data in the target domain, so that the model has good generalization performance and can be widely used in practical environments.

[0008] In order to achieve the above object, the content of the present invention comprises the following steps:

[0009] Step 1: Dataset preprocessing;

[0010] Step 2: Pre-train the ResNet-50 pre-trained on ImageNet using the source domain dataset to obtain a baseline model.

[0011] Step 3: Combine the source domain dataset and the target domain dataset to form a joint dataset, and use the joint dataset to train and optimize the baseline model;

[0012] Step 4: Test model performance;

[0013] Step 1: The specific method is as follows;

[0014] We use labeled source domain datasets and unlabeled target domain datasets, where the source domain dataset is denoted as Ds, the total number of source domain images is denoted as Ns, the total number of IDs is denoted as Ms, and each sample is denoted as The corresponding label is The target domain dataset is denoted as Dt, the total number of source domain images is denoted as Nt, and each sample is denoted as Data enhancement is performed on the above dataset, including randomly selecting images in the dataset for flipping, cropping, and erasing.

[0015] Step 2: The specific method is as follows;

[0016] Select ResNet-50 pre-trained on ImageNet as the backbone network, input the source domain dataset to complete pre-training in the source domain, and use the classification loss function and the triplet loss function to train the baseline model together. The total loss function is:

[0017]

[0018] where λ s is a hyperparameter used to balance the two loss functions.

[0019] Classification loss function The specific expression is as follows:

[0020]

[0021] in is the cross entropy loss function, F(·|θ) is the backbone network, Output features for the network.

[0022] Triplet loss function The specific expression is as follows

[0023]

[0024] where ||·|| is the Euclidean distance, and They are The positive and negative samples are , and m is the distance margin.

[0025] Step 3: The specific method is as follows;

[0026] The training process combines the source domain dataset and the target domain dataset to form a joint dataset, denoted as in Corresponding to the source domain dataset, Corresponding to the target domain dataset, N = N s +N t , the combined dataset Input into the baseline model trained in step (2) to obtain the corresponding feature representation, denoted as in

[0027] In order to achieve the goal of joint training, a hybrid memory module is used to store the joint dataset. The feature representation of is obtained by taking the feature representation of and updating the feature representation in the hybrid memory module at the beginning of each epoch. The hybrid memory module is denoted as Its size is d×N, and For the joint dataset Any sample in Its characteristics are expressed as It is different from any other sample in the dataset The similarity is expressed by cosine distance, and the specific calculation formula is:

[0028]

[0029] According to S i,j Select samples whose values ​​are greater than the similarity confidence t and sort them according to the similarity S i,jThe positive sample set P is formed by sorting them from large to small, and the number of samples in the set P is recorded as k, based on which multi-category labels are generated for joint training.

[0030] When the sample When it comes from the source domain dataset, the number of samples in the positive sample set P that also come from the source domain dataset is counted and recorded as a, and the corresponding multi-category label is If there are samples from the target domain dataset in the positive sample set, the corresponding multi-category labels are For the joint dataset The multi-category labels corresponding to the remaining samples except the positive sample set P are The specific calculation formula is as follows:

[0031]

[0032] If the sample From the target domain dataset, the multi-category labels corresponding to the samples in the positive sample set P are Joint Datasets The multi-category labels corresponding to the samples other than the positive sample set P are The specific calculation formula is as follows:

[0033]

[0034] Based on the multi-category labels generated above, the present invention proposes

[0035] Multi-Label-Cross-Entropy Loss (MLCEL), the specific calculation formula of MLCEL is as follows

[0036]

[0037] The loss function calculates the corresponding loss functions according to the classification of positive samples and negative samples, and uses parameters λ1 and λ2 to balance the proportion of the two types of loss functions in MLCEL, so as to obtain better experimental performance.

[0038] In addition, the hybrid memory updates the feature representation of the stored dataset image at the beginning of each epoch. The specific calculation formula is as follows

[0039]

[0040] Where t represents the current epoch, α is the memory update rate, After that, L2 normalization is required, and the calculation formula is as follows

[0041]

[0042] Combine the datasets Input into the baseline model, use MLCEL to train and optimize the baseline model, and obtain a model with good performance and strong generalization ability.

[0043] Step 4: The specific method is as follows;

[0044] The images in the query set and gallery set are input into the model trained in step (3) to obtain the feature representation of the image. The Euclidean distance between the image features in the query and the image features in the gallery is calculated respectively. The images are sorted according to the calculated Euclidean distance and the closest image with the smallest Euclidean distance is selected as the recognition result. The mAP and CMC are calculated respectively to evaluate the model performance.

[0045] The advantages and beneficial results of the method of the present invention are:

[0046] (1) By pre-training the backbone network on ImageNet and the source domain, a baseline with good model performance can be obtained, providing better feature representation for subsequent processing;

[0047] (2) By jointly training the source domain and target domain datasets, we can further utilize the rich annotation information in the source domain dataset and further narrow the domain differences, thereby achieving better adaptation effects;

[0048] (3) By generating multi-category labels based on similarity and hybrid memory, similar images can be searched in both the source and target domains, further improving the performance of the model and avoiding clustering noise similar to that in clustering-based algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Flow chart of an embodiment of the method of the present invention. DETAILED DESCRIPTION

[0050] The present invention is described in detail below in conjunction with specific implementation modes.

[0051] like Figure 1 As shown, the hybrid memory-based joint training cross-domain pedestrian re-identification method proposed in the present invention is implemented according to the following steps.

[0052] Step 1: Dataset preprocessing;

[0053] We use a labeled source domain dataset and an unlabeled target domain dataset, where the source domain dataset is Market1501, denoted as Ds, and the total number of source domain images is denoted as N. s =1293, the total number of IDs is recorded as M s =751, where each sample is recorded as The corresponding label is The target domain dataset is DukeMTMC-reID, denoted as Dt, and the total number of source domain images is denoted as N t =702, where each sample is recorded as Data enhancement is performed on the above dataset, including randomly selecting several images in the dataset for flipping, cropping, and erasing.

[0054] Step 2: Pre-training in the source domain;

[0055] We select ResNet-50 pre-trained on ImageNet as the backbone network, input the source domain dataset to complete pre-training in the source domain, and use the classification loss function and the triplet loss function to train the baseline model together. The total loss function is:

[0056]

[0057] where λ s is a hyperparameter used to balance the two loss functions.

[0058] Classification loss function The specific expression is as follows

[0059]

[0060] in is the cross entropy loss function, F(·|θ) is the backbone network, Output features for the network.

[0061] Triplet loss function The specific expression is as follows

[0062]

[0063] where ||·|| is the Euclidean distance, and They are The positive and negative samples are , and m is the distance margin.

[0064] Step 3: Joint training in source and target domains

[0065] The training process combines the source domain dataset and the target domain dataset to form a joint dataset, denoted as in Corresponding to the source domain dataset, Corresponding to the target domain dataset, N = N s +N t , the combined dataset Input into the baseline model trained in step (2) to obtain the corresponding feature representation, denoted as in

[0066] In order to achieve the goal of joint training, the present invention uses a hybrid memory module to store the joint data set The feature representation of , and update the feature representation in the memory at the beginning of each epoch. The hybrid memory module is denoted as Its size is d×N, and For the joint dataset Any sample in Its characteristics are expressed as It is different from any other sample in the dataset The similarity is expressed by cosine distance, and the specific calculation formula is:

[0067]

[0068] According to S i,j Select samples whose values ​​are greater than the similarity confidence t and sort them according to the similarity S i,j The positive sample set P is formed by sorting them from large to small, and the number of samples in the set P is recorded as k, based on which multi-category labels are generated for joint training.

[0069] If the sample From the source domain dataset, count the number of samples in the positive sample set P that also come from the source domain dataset and record it as a, and the corresponding multi-category labels are If there are samples from the target domain dataset in the positive sample set, the corresponding multi-category labels are For the joint dataset The multi-category labels corresponding to the remaining samples except the positive sample set P are The specific calculation formula is as follows

[0070]

[0071] If the sample From the target domain dataset, the multi-category labels corresponding to the samples in the positive sample set P are Joint Datasets The multi-category labels corresponding to the samples other than the positive sample set P are The specific calculation formula is as follows

[0072]

[0073] Based on the multi-category labels generated above, the present invention proposes Multi-Label-Cross-Entropy Loss (MLCEL). The specific calculation formula of MLCEL is as follows:

[0074]

[0075] The loss function calculates the corresponding loss functions according to the classification of positive samples and negative samples, and uses parameters λ1 and λ2 to balance the proportion of the two types of loss functions in MLCEL, so as to obtain better experimental performance.

[0076] In addition, the hybrid memory updates the feature representation of the stored dataset image at the beginning of each epoch. The specific calculation formula is as follows

[0077]

[0078] Where t represents the current epoch, α is the memory update rate, After that, L2 normalization is required, and the calculation formula is as follows

[0079]

[0080] Combine the datasets Input into the baseline model, use MLCEL to train and optimize the baseline model, and obtain a model with good performance and strong generalization ability.

[0081] Step 4: Test model performance;

[0082] The images in the query set and gallery set are input into the model trained in step (3) to obtain the feature representation of the image. The Euclidean distance between the image features in the query and the image features in the gallery is calculated respectively. The images are sorted according to the calculated Euclidean distance and the closest image with the smallest Euclidean distance is selected as the recognition result. The mAP and CMC are calculated respectively to evaluate the model performance.

Claims

1. A cross-domain person re-identification method based on joint training of hybrid memory, characterized in that: The following steps are involved: Step 1: Dataset preprocessing; Step 2: Pre-train the ResNet-50 pre-trained on ImageNet using the source domain dataset to obtain a baseline model. Step 3: Combine the source domain dataset and the target domain dataset to form a joint dataset, and use the joint dataset to train and optimize the baseline model; Step 4: Test model performance; Step 1: The specific method is as follows; We use labeled source domain datasets and unlabeled target domain datasets, where the source domain dataset is denoted as Ds, the total number of source domain images is denoted as Ns, the total number of IDs is denoted as Ms, and each sample is denoted as The corresponding label is The target domain dataset is denoted as Dt, the total number of source domain images is denoted as Nt, and each sample is denoted as Perform data augmentation on the above dataset, including randomly selecting images in the dataset for flipping, cropping, and erasing; Step 2: The specific method is as follows; Select ResNet-50 pre-trained on ImageNet as the backbone network, input the source domain dataset to complete pre-training in the source domain, and use the classification loss function and the triplet loss function to train the baseline model together. The total loss function is: where λ s is a hyperparameter used to balance the two loss functions; Classification loss function The specific expression is as follows: in is the cross entropy loss function, F(·|θ) is the backbone network, Output features for the network; Triplet loss function The specific expression is as follows where ||·|| is the Euclidean distance, and They are Positive and negative samples, m is the distance margin; Step 3: The specific method is as follows; The training process combines the source domain dataset and the target domain dataset to form a joint dataset, denoted as in Corresponding to the source domain dataset, Corresponding to the target domain dataset, N = N s +N t , the combined dataset Input into the baseline model trained in step (2) to obtain the corresponding feature representation, denoted as in In order to achieve the goal of joint training, a hybrid memory module is used to store the joint dataset. The feature representation of , and update the feature representation in the hybrid memory module at the beginning of each epoch; the hybrid memory module is denoted as Its size is d×N, and For the joint dataset Any sample in Its characteristics are expressed as It is different from any other sample in the dataset The similarity is expressed by cosine distance, and the specific calculation formula is: According to S i,j Select samples whose values ​​are greater than the similarity confidence t and sort them according to the similarity S i,j Sort the samples from large to small to form a positive sample set P. The number of samples in set P is recorded as k, and multi-category labels are generated based on this for joint training. When the sample When it comes from the source domain dataset, the number of samples in the positive sample set P that also come from the source domain dataset is counted and recorded as a, and the corresponding multi-category labels are If there are samples from the target domain dataset in the positive sample set, the corresponding multi-category labels are For the joint dataset The multi-category labels corresponding to the remaining samples except the positive sample set P are The specific calculation formula is as follows: If the sample From the target domain dataset, the multi-category labels corresponding to the samples in the positive sample set P are Joint Datasets The multi-category labels corresponding to the samples other than the positive sample set P are The specific calculation formula is as follows: Based on the multi-category labels generated above, Multi-Label-Cross-Entropy Loss (MLCEL) is proposed. The specific calculation formula of MLCEL is as follows The loss function calculates the corresponding loss functions according to the classification of positive samples and negative samples, and uses parameters λ1 and λ2 to balance the proportion of the two types of loss functions in MLCEL, so as to obtain better experimental performance; In addition, the hybrid memory updates the feature representation of the stored dataset image at the beginning of each epoch. The specific calculation formula is as follows Where t represents the current epoch, α is the memory update rate, After that, L2 normalization is required, and the calculation formula is as follows Combine the datasets Input into the baseline model, use MLCEL to train and optimize the baseline model, and obtain a model with good performance and strong generalization ability.

2. According to the hybrid memory-based joint training cross-domain pedestrian re-identification method of claim 1, it is characterized in that: Step 4: The specific method is as follows; The images in the query set and gallery set are input into the model trained in step (3) to obtain the feature representation of the image. The Euclidean distance between the image features in the query and the image features in the gallery is calculated respectively. The images are sorted according to the calculated Euclidean distance and the closest image with the smallest Euclidean distance is selected as the recognition result. The mAP and CMC are calculated respectively to evaluate the model performance.