A Deep Learning-Based Multi-Source Domain Person Re-identification Domain Generalization Method

Abstract

This invention discloses a deep learning-based multi-source domain person re-identification domain generalization method, comprising the following steps: preparing a dataset; training multiple expert models using a cyclic serial training method; constructing a loss function using the idea of ​​hard sample mining; incorporating a lifelong learning strategy during the serial training process; in the testing phase, performing a weighted average of all expert models to obtain the final model; organizing the pedestrian target images to be detected in the target domain, selecting images with more features as query images and inputting them into the trained model; using all pedestrian sample images generated from surveillance videos as the detection set for detection and matching; the model outputs pedestrian samples with high matching degrees, arranged from high to low similarity; the operator finds the same-identity pedestrian target in the pedestrian target list and performs pedestrian tracking. This invention solves the problems of memory explosion and mutual interference between multiple source domains.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically to a deep learning-based multi-source domain person re-identification domain generalization method. Background Technology

[0002] Pedestrian re-identification (ReID)[1] refers to the identification and tracking of pedestrians in multiple scenes through computer vision technology, thereby realizing the tracking and querying of pedestrian identities. Its significance lies in its applicability to many fields, such as security, intelligent transportation, and pedestrian flow statistics. Traditional supervised ReID is designed for test data to come from the same distribution as the training data and has achieved significant performance. However, these supervised ReID methods are designed based on a single data distribution, and are therefore limited by the out-of-domain generalization ability of the model, making it difficult to apply to real-world scenarios. Therefore, domain generalized pedestrian re-identification (DG-ReID)[2] has gradually attracted widespread research attention. Given a set of labeled source domains, DG-ReID aims to train a model that can perform well in unseen domains without fine-tuning. Currently, there are CUHK03, MARKET1501, MSMT17, DUKEMTMC, etc. on domain generalized pedestrian re-identification datasets.

[0003] Currently, the mainstream approaches to generalized person re-identification are all based on the same baseline. This involves combining all source datasets into a single dataset and training the model directly on this dataset using supervised person re-identification methods. The differences between these mainstream approaches lie in the choice of network architecture [3,4,5], training strategy [6,7], and data augmentation schemes [8,9]. The problem with this baseline approach is that it doesn't adequately consider the large number of pedestrians (i.e., categories) resulting from combining images from each dataset, leading to extremely high memory consumption and posing a significant challenge to hardware implementation during training. More importantly, because multiple datasets are trained together, the domain differences between them can be substantial, causing conflicts between datasets during training. This results in poor generalization performance even in the source domain, limiting the model's applicability.

[0004] References:

[0005] [1] N.Gheissari, TB Sebastian, and R.Hartley, "Person reidentification using spatiotemporal appearance," in CVPR, vol.2, 2006, pp.1528-1535.

[0006] [2]J.Song,Y.Yang,Y.-Z.Song,T.Xiang,and T.M.Hospedales,“Generalizableperson re-identification by domain invariant mapping network,”in CVPR,2019,pp.719-728.

[0007] [3]J.Jia,Q.Ruan,and T.M.Hospedales,“Frustratingly easy person re-identification:Generalizing person re-id in practice,”in BMVC,2019,p.117.

[0008] [4]J.Liu,Z.Huang,L.Li,K.Zheng,and Z.-J.Zha,“Debiased batchnormalization via gaussian process for generalizable person re-identification,”in AAAI,2022,pp.1729-1737.

[0009] [5]X.Jin,C.Lan,W.Zeng,Z.Chen,and L.Zhang,“Style normalization andrestitution for generalizable person re-identification,”in CVPR,2020,pp.3143-3152

[0010] [6]Y.Zhao,Z.Zhong,F.Yang,Z.Luo,Y.Lin,S.Li,and N.Sebe,“Learning togeneralize unseen domains via memory-based multi-source meta-learning forperson re-identification,”in CVPR,2021,pp.6277-6286.

[0011] [7] K.Han, C.Si, Y.Huang, L.Wang, and T.Tan, "Generalizable person re-identification via self-supervised batch norm test-time adaption," in AAAI, 2022, pp.817-825.

[0012] [8] K. Zhou, Y. Yang, Y. Qiao, and T. Xiang, “Domain generalization with mixstyle,” in ICLR, 2021.

[0013] [9] X. Li, Y. Dai, Y. Ge, J. Liu, Y. Shan, and L.-Y. Duan, “Uncertainty modeling for out-of-distribution generalization,” in ICLR, 2022. Summary of the Invention

[0014] The purpose of this invention is to provide a deep learning-based multi-source domain pedestrian re-identification domain generalization method to solve the problems of memory explosion and mutual interference between multi-source domains in the prior art.

[0015] To achieve the above object, the present invention adopts the following technical solutions:

[0016] A deep learning-based multi-source domain person re-identification domain generalization method includes the following steps:

[0017] Step 1: Prepare the dataset and preprocess it;

[0018] Step 2: Train multiple expert models using a cyclical, sequential training method;

[0019] Step 3: Using the idea of ​​hard sample mining, a loss function is constructed so that each expert model can comprehensively consider the differences between different cameras during the training process.

[0020] Step 4, Style Replay: Incorporate lifelong learning strategies into the sequential training process;

[0021] Step 5, Testing: In the testing phase, a weighted average is calculated for all expert models to obtain the final model;

[0022] Step 6: Organize the pedestrian target images to be detected in the target domain, select images with more features as query images and input them into the weighted average model, use all pedestrian sample images generated by surveillance video as the detection set for detection and matching, the model will output pedestrian samples with high matching degree, arranged from high to low similarity, the operator finds the same identity pedestrian target in the pedestrian target list and performs pedestrian tracking.

[0023] In step 1, the dataset covers different locations and indoor / outdoor conditions.

[0024] In step 1, the preprocessing steps for the dataset are as follows: images without pedestrians are removed; in addition, for images containing pedestrians, object detection methods are used to detect pedestrians, and the corresponding detection boxes are extracted for subsequent training.

[0025] In step 2, when training multiple expert models using a cyclic serial training method, each batch of data comes from only one source dataset, and the data is sampled from a single dataset for a continuous period of time.

[0026] In step 3, the loss function links the importance of each camera to the similarity between the camera's centroid and the current training instance. This loss function assigns greater weight to less similar positive camera centroids and more similar negative camera centroids. The specific formula is as follows:

[0027]

[0028] Where x is a pedestrian image, assuming it is the Lth pedestrian in the dataset, captured by camera C, and f represents an expert model being trained. x =f θ (x) is the feature vector output by x after passing through the model, where δ and λ1 are hyperparameters, and l represents the l-th pedestrian. The similarity between the feature vector of a given image x of the current pedestrian and the feature vector of the l-th pedestrian is expressed by the formula:

[0029]

[0030] in, It is f x The transpose of τ is a hyperparameter. This represents the center coordinates of the l-th pedestrian under camera c.

[0031] In step 4, the lifelong learning strategy adopts the idea of ​​a style library. During the serial training process, the lifelong learning strategy samples a style from the style library of a training domain and merges it with the style of the source domain sample currently being trained, so that the expert of the current training domain can generalize to the source domain that has been trained before.

[0032] In step 4, let the current training domain be B, and the source domain A have already been trained; the network model f used is ResNet50, which contains 4 residual modules, i.e., f = [res1, res2, res3, res4]; let g = [res1, res2], for a given... Let x be the output of an intermediate layer of network model f, and g be a part of network f, where x b This is an image batch from domain B, with a batch size of B. s C, H, and W represent the number of channels, height, and width of the intermediate layer feature, respectively; and These represent the mean and standard deviation of the channel features for each instance and each mini-batch, respectively; their calculation methods are as follows:

[0033]

[0034]

[0035]

[0036]

[0037] Where b, c, h, and w represent the b-th image, the c-th channel, and the height and width coordinates of v, respectively.

[0038] definition:

[0039]

[0040]

[0041] The style library of domain B is defined as two multivariate Gaussian distributions. and Style (μ) B , σ B Sampling is performed from the two distributions mentioned above, i.e.

[0042] Based on the above definition, suppose domain B is being trained and domain A has already been trained. To ensure good model performance in domain A, image information from A is needed; styles (μ) are randomly sampled from the style library of domain A. A , σ A) is mixed with the current feature v to synthesize a new feature.

[0043]

[0044]

[0045]

[0046] Where m is a hyperparameter, and finally... It will replace v and be sent into the subsequent network.

[0047] In step 5, during the testing phase, the similarity measurement method based on bulldozer distance saves the feature vector of each pedestrian in each source domain and uses it to calculate the bulldozer distance between each source domain and the target domain. Then, based on the distance, the weight is calculated using the softmax function as the importance of each source domain and source domain expert. The smaller the bulldozer distance, the higher the importance.

[0048] In step 5, at the last N S In each epoch, at the end of each epoch, the model's parameters are stored as experts in the current domain. For the k-th domain, its experts are represented as... Simultaneously, the MCM of each source domain will be stored, and the MCM of the k-th domain will be represented as... Where, N S Where epoch represents the number of source domains, MCM represents the identity center vector library;

[0049] The testing process is as follows:

[0050] (1) Test domain D U Gallery (G) U All images in the dataset are sent to the k-th domain expert. Perform forward pass to obtain

[0051] (2) Calculation With the j-th source domain memory The Wasserstein distance between them is expressed as Intuitively, It should be relatively close to the k-th source region and relatively far from other source regions; therefore, the unseen region D... U The similarity with the k-th source domain is defined as:

[0052]

[0053] (3) Then apply the SoftMax function to To obtain the weights, the formula is:

[0054]

[0055] After obtaining the weights, for the test domain D U For any image x, its characteristic feature vector is represented as:

[0056]

[0057] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0058] (1) This invention employs a cyclic serial training method to train multiple expert models, which significantly reduces memory consumption and effectively alleviates the memory explosion problem. Serial training can isolate the training between different source domains, thereby mitigating the problem of mutual interference.

[0059] (2) The newly proposed loss based on hard sample mining in this invention enables the model to comprehensively consider the differences between different cameras during the training process.

[0060] (3) The new lifelong learning strategy based on style library replay of the present invention enables experts in the current training domain to generalize to the source domain that have been trained before, thereby alleviating the phenomenon of learning and forgetting.

[0061] (4) The similarity (weight) calculation method of the present invention enables source domains that are similar to the target domain to play a greater role in the testing process.

[0062] (5) The present invention achieves state-of-the-art results on two evaluation frameworks and a total of eight datasets. Attached Figure Description

[0063] Figure 1 This is a comparison chart of traditional parallel training and the serial loop training proposed in this invention.

[0064] Figure 2 This invention proposes a lifelong learning strategy called style replay. Detailed Implementation

[0065] The invention will now be further explained with reference to the accompanying drawings.

[0066] The present invention provides a deep learning-based multi-source domain person re-identification domain generalization method, comprising the following steps:

[0067] Step 1, Dataset Preparation: The raw data for pedestrian re-identification is usually surveillance video, which requires manual or algorithmic extraction of key pedestrian information. For example, Faster R-CNN can be used to crop pedestrian images from surveillance video frames. This invention uses four cropped datasets—CUHK03, MARKET1501, MSMT17, and DUKEMTMC—as experimental datasets.

[0068] Step 2, Cyclic training: such as Figure 1 As shown, this invention abandons the traditional multi-source domain hybrid parallel training method and adopts a cyclic serial training method to train multiple expert models. Specifically, in traditional multi-source domain training, the data in each batch is uniformly sampled from different datasets, and each batch contains data from different source datasets. In this invention, each batch of data comes from only one source dataset, and the data is sampled from only one dataset for a continuous period of time. Compared to the traditional method, which needs to store the centroids of all source datasets in each epoch, this invention only needs to store the centroid of only one source dataset in each epoch, which greatly reduces the required GPU memory consumption.

[0069] Step 3, Hard Sample Mining Loss: Specifically, for a training instance, this loss function assigns greater weight to centroids of less similar positive cameras and more similar negative cameras. The specific formula is as follows:

[0070]

[0071] Where x is a pedestrian image, assuming it is the Lth pedestrian in the dataset, captured by camera C, and f represents an expert model being trained. x =f θ (x) is the feature vector output by x after passing through the model, where δ and λ1 are hyperparameters, and l represents the l-th pedestrian. The similarity between the feature vector of a given image x of the current pedestrian and the feature vector of the l-th pedestrian is expressed by the formula:

[0072]

[0073] in, It is f x The transpose of τ is a hyperparameter. This represents the center coordinates of the l-th pedestrian under camera c.

[0074] Step 4, style replay: such as Figure 2As shown, in the strict definition of lifelong learning, each source domain can only be trained once. However, in domain generalization tasks, there is no such restriction. Therefore, the simplest way to alleviate the catastrophic forgetting problem is to repeat the serial training process multiple times. This is why "Recurrent Serial Training" was proposed, which significantly addresses the catastrophic forgetting problem. In addition to Recurrent Serial Training, some rigorous lifelong learning methods can be incorporated as supplementary approaches. However, integrating traditional lifelong learning methods into a framework may not be a cost-effective solution for two reasons. First, the catastrophic forgetting problem has already been significantly addressed by Recurrent Serial Training. Second, since these methods are specifically designed for catastrophic forgetting and adhere to the requirement of training only once, additional computational effort is unavoidable. Therefore, Style Replay was proposed, which requires minimal computational effort.

[0075] Let the current training domain be B, and the source domain A have already been trained; the network model f used is ResNet50, which contains 4 residual modules, i.e., f = [res1, res2, res3, res4]; let g = [res1, res2], for a given... Let x be the output of an intermediate layer of network model f, and g be a part of network f, where x b This is an image batch from domain B, with a batch size of B. s C, H, and W represent the number of channels, height, and width of the intermediate layer feature, respectively; and These represent the mean and standard deviation of the channel features for each instance and each mini-batch, respectively; their calculation methods are as follows:

[0076]

[0077]

[0078]

[0079]

[0080] Where b, c, h, and w represent the b-th image, the c-th channel, and the height and width coordinates of v, respectively.

[0081] Furthermore, define:

[0082]

[0083]

[0084] The style library of domain B is defined as two multivariate Gaussian distributions. and Style (μ) B , σ B Sampling can be performed from the two distributions mentioned above, i.e.

[0085] Based on the above definition, assuming domain B is being trained and domain A has already been trained, to ensure good model performance on domain A, image information from A is needed. One forward solution is to randomly sample styles (μ) from the style library of domain A. A , σ A ) is mixed with the current feature v to synthesize a new feature.

[0086]

[0087]

[0088]

[0089] Finally, m is a hyperparameter, which is set to 0.1 in this embodiment. It will replace v and be sent into the subsequent network.

[0090] Step 5, Testing: During the testing phase, the similarity measurement method based on bulldozer distance saves the feature vector of each pedestrian in each source domain, and uses this to calculate the bulldozer distance between each source and target domain. Then, based on the distance, a weight is calculated using the softmax function as the importance of each source domain and source domain expert; the smaller the bulldozer distance, the higher the importance. Finally, N... S In (number of source domains) epochs, at the end of each epoch, the model's parameters will be stored as experts in the current domain. For the k-th domain, its experts will be represented as... Simultaneously, the MCM (Identity Center Vector Library) for each source domain will be stored, and the MCM for the k-th domain will be represented as...

[0091] The testing process is as follows:

[0092] (1) Test domain D U Gallery (G) U All images in the dataset are sent to the k-th domain expert. Perform forward pass to obtain

[0093] (2) Calculation With the j-th source domain memory The Wasserstein distance between them is expressed as Intuitively, It should be relatively close to the k-th source region and relatively far from other source regions. Therefore, the unseen region D... U The similarity with the k-th source domain is defined as:

[0094]

[0095] (3) Then apply the SoftMax function to To obtain the weights. The formula is:

[0096]

[0097] In the experiment, ∈ was set to 0.005. After obtaining the weights, for the test domain D... U For any image x (including Gallery and Query images), its characteristic feature vector can be represented as:

[0098]

[0099] Step 6: Organize the pedestrian target images to be detected in the target domain, select images with more features as query images and input them into the trained model, use all pedestrian sample images generated from surveillance videos as the detection set for detection and matching, the model will output pedestrian samples with high matching degree, arranged from high to low similarity, the operator finds the same pedestrian target according to the pedestrian target list and performs pedestrian tracking.

[0100] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A deep learning-based multi-source domain person re-identification domain generalization method, characterized in that: Includes the following steps: Step 1: Prepare the dataset and preprocess it; Step 2: Train multiple expert models using a cyclical, sequential training method; Step 3: Using the idea of ​​hard sample mining, a loss function is constructed so that each expert model can comprehensively consider the differences between different cameras during the training process. Step 4, Style Replay: Incorporate lifelong learning strategies into the sequential training process; Let the current training domain be B, and the source domain A have already been trained; the network model f used is ResNet50, which contains 4 residual modules, i.e., f = [res1, res2, res3, res4]; let g = [res1, res2], for a given... Let x be the output of an intermediate layer of network model f, and g be a part of network f, where x b This is an image batch from domain B, with a batch size of B. s C, H, and W represent the number of channels, height, and width of the intermediate layer feature, respectively; and These represent the mean and standard deviation of the channel features for each instance and each mini-batch, respectively; their calculation methods are as follows: Where b, c, h, w represent the b-th image, the c-th channel, and the height and width coordinates of v, respectively. definition: The style library of domain B is defined as two multivariate Gaussian distributions. and Style (μ) B ,σ B Sampling is performed from the two distributions mentioned above, i.e. Based on the above definition, suppose domain B is being trained and domain A has already been trained. To ensure good model performance in domain A, image information from A is needed; styles (μ) are randomly sampled from the style library of domain A. A ,σ A ) is mixed with the current feature v to synthesize a new feature. Where m is a hyperparameter, and finally... It will replace v and be sent into the subsequent network; Step 5, Testing: In the testing phase, a weighted average is calculated for all expert models to obtain the final model; Step 6: Organize the pedestrian target images to be detected in the target domain, select images with more features as query images and input them into the weighted average model, use all pedestrian sample images generated by surveillance video as the detection set for detection and matching, the model will output pedestrian samples with high matching degree, arranged from high to low similarity, the operator finds the same identity pedestrian target in the pedestrian target list and performs pedestrian tracking.

2. The deep learning-based multi-source domain person re-identification domain generalization method according to claim 1, characterized in that: In step 1, the dataset covers different locations and indoor / outdoor conditions.

3. The deep learning-based multi-source domain person re-identification domain generalization method according to claim 1, characterized in that: In step 1, the preprocessing steps for the dataset are as follows: images without pedestrians are removed; in addition, for images containing pedestrians, object detection methods are used to detect pedestrians, and the corresponding detection boxes are extracted for subsequent training.

4. The multi-source domain person re-identification domain generalization method based on deep learning according to claim 1, characterized in that: In step 2, when training multiple expert models using a cyclic serial training method, each batch of data comes from only one source dataset, and the data is sampled from a single dataset for a continuous period of time.

5. The multi-source domain person re-identification domain generalization method based on deep learning according to claim 1, characterized in that: In step 3, the loss function links the importance of each camera to the similarity between the camera's centroid and the current training instance. This loss function assigns greater weight to less similar positive camera centroids and more similar negative camera centroids. The specific formula is as follows: Where x is a pedestrian image, assuming it is the Lth pedestrian in the dataset, captured by camera C, and f represents an expert model being trained. x =f θ (x) is the feature vector output by x after passing through the model, where δ and λ1 are hyperparameters, and l represents the l-th pedestrian. The similarity between the feature vector of a given image x of the current pedestrian and the feature vector of the l-th pedestrian is expressed by the formula: in, It is f x The transpose of τ is a hyperparameter. This represents the center coordinates of the l-th pedestrian under camera c.

6. The multi-source domain person re-identification domain generalization method based on deep learning according to claim 1, characterized in that: In step 4, the lifelong learning strategy adopts the idea of ​​a style library. During the serial training process, the lifelong learning strategy samples a style from the style library of a training domain and merges it with the style of the source domain sample currently being trained, so that the expert of the current training domain can generalize to the source domain that has been trained before.

7. The multi-source domain person re-identification domain generalization method based on deep learning according to claim 1, characterized in that: In step 5, during the testing phase, the similarity measurement method based on bulldozer distance saves the feature vector of each pedestrian in each source domain and uses it to calculate the bulldozer distance between each source domain and the target domain. Then, based on the distance, the weight is calculated using the softmax function as the importance of each source domain and source domain expert. The smaller the bulldozer distance, the higher the importance.

8. The deep learning-based multi-source domain person re-identification domain generalization method according to claim 1 or 7, characterized in that: In step 5, at the last N S In each epoch, at the end of each epoch, the model's parameters are stored as experts in the current domain. For the k-th domain, its experts are represented as... Simultaneously, the MCM of each source domain will be stored, and the MCM of the k-th domain will be represented as... Where, N S Where epoch represents the number of source domains, MCM represents the identity center vector library; The testing process is as follows: (1) Test domain D U Gallery (G) U All images in the dataset are sent to the k-th domain expert. Perform forward pass to obtain (2) Calculation With the j-th source domain memory The Wasserstein distance between them is expressed as It is relatively close to the k-th source region and relatively far from other source regions; therefore, the unseen region D is... U The similarity with the k-th source domain is defined as: (3) Then apply the SoftMax function to To obtain the weights, the formula is: After obtaining the weights, for the test domain D U For any image x, its characteristic feature vector is represented as:

Images