Visible light and infrared cross-modal pedestrian re-identification method adaptive to unsupervised domain

Abstract

The invention discloses a visible light and infrared cross-modal pedestrian re-identification method adaptive to an unsupervised domain, and belongs to the technical field of cross-modal pedestrian re-identification. The method comprises the following steps: 1, constructing a visible light and infrared single-mode pedestrian re-identification single-mode model adaptive to an unsupervised domain; and 2, training a target domain sample and a source domain class prototype by adopting the visible light and infrared single-mode pedestrian re-identification model adaptive to the unsupervised domain in the step 1. And 3, using a soft pseudo label of a target domain sample trained by the visible light and infrared single-mode pedestrian re-identification model adapted to the non-supervised domain in the step 2 for visible light and infrared cross-mode pedestrian re-identification training adapted to the non-supervised domain. Compared with the prior art, the generalization ability of visible light and infrared cross-modal pedestrian re-identification adapted to the unsupervised domain is improved.

Description

Technical Field

[0001] The present invention relates to an unsupervised domain adaptation visible light and infrared cross-modal pedestrian re-identification method, belonging to the technical field of cross-modal pedestrian re-identification. Background Art

[0002] Pedestrian re-identification technology aims to accurately match the images of pedestrians under different cameras from a database containing a large number of pedestrian images, so as to solve the problem of pedestrian matching across cameras and scenes. Pedestrian re-identification is mainly divided into single-modal pedestrian re-identification and visible light and infrared cross-modal pedestrian re-identification, which is of great significance for realizing all-weather video surveillance, intelligent security, and crowd flow analysis.

[0003] Currently, the research methods of visible light and infrared cross-modal pedestrian re-identification are divided into supervised learning, unsupervised learning and unsupervised domain adaptation.

[0004] Supervised learning-based methods are performed on labeled datasets and have achieved good results. However, due to the high cost of obtaining high-quality annotations, the application scenarios of such methods are relatively limited.

[0005] Visible-infrared cross-modal person re-identification based on unsupervised learning is a more practical solution that aims to study the relationship between the data itself without relying on external labels.

[0006] Methods based on unsupervised domain adaptation have been widely used in pedestrian re-identification. The knowledge learned from another related field with labeled data is used to improve the performance of the model in the target field with unlabeled data. However, in visible light-infrared cross-modal pedestrian re-identification, due to the modal differences within the domain, the existing unsupervised domain adaptation methods are difficult to simultaneously solve the distribution differences between domains and the modal differences within the domain, resulting in poor generalization ability of the model.

[0007] Therefore, how to apply unsupervised domain adaptation methods to improve generalization ability in pedestrian re-identification has become an urgent problem to be solved. Summary of the invention

[0008] The purpose of the present invention is to solve the technical problem of applying unsupervised domain adaptation methods to improve the generalization ability in pedestrian re-identification, and to propose an unsupervised domain adaptation visible light and infrared cross-modal pedestrian re-identification method.

[0009] The objective of the present invention is achieved through the following technical solutions:

[0010] The present invention provides an unsupervised domain adaptation visible light and infrared cross-modal pedestrian re-identification method, comprising the following steps:

[0011] Step 1: Build a unimodal model for visible light and infrared unimodal person re-identification with unsupervised domain adaptation.

[0012] Step 1.1: Label the target domain dataset to be trained using a camera-driven method to form target domain training samples with camera labels.

[0013] Step 1.2: According to step 1.1, the target domain training samples and the source domain are sorted in a monotonically increasing order using the distribution difference shown in formula (1).

[0014]

[0015] Among them, D s , represents the source domain and the target domain, n∈{1,...,N} is the camera number, φ(·) represents the Gaussian kernel function, represents the reproducing Hilbert space, and |·| represents the total number of samples in the dataset.

[0016] Step 1.3: Classify the labeled samples in the source domain in visible light mode and infrared mode, and extract features from the source domain samples and perform calculations as shown in formula (2). Thus, the feature mean of the source domain is obtained, which is the class prototype of the source domain.

[0017]

[0018] Among them, k∈{1,...,C} is the label of the class contained in the source domain, They represent the class prototypes of the kth class in the visible light modality and infrared modality in the source domain, Represent the number of samples of the kth class in the visible light modality and infrared modality respectively.

[0019] Step 2: Use the unsupervised domain adapted visible light and infrared single-modal pedestrian re-identification model described in step 1 to train the target domain samples and source domain class prototypes, so as to improve the generalization ability of unsupervised domain adapted visible light and infrared single-modal pedestrian re-identification.

[0020] Step 2.1: Construct the target domain sample feature pairs so that the target domain samples correspond to the source domain class prototypes in the form of labels.

[0021] Step 2.1.1: Get the label of the prototype of the source domain class with the smallest distance.

[0022] Step 2.1.2: Assign the label of the source domain class prototype in step 2.1.1 to the target domain sample to form a hard pseudo label of the target domain sample as shown in formula (3).

[0023]

[0024] in, Respectively represent the labels of the smallest source domain prototype visible light modality and infrared modality, denote the visible light modality samples and infrared modality samples of the target domain respectively, min(·) denotes the label of the minimum source domain class prototype, and the cosine distance between the visible light modality and infrared modality features of the target domain samples and the source domain class prototype is calculated to obtain the label of the source domain class prototype with the minimum distance and assign it to the target domain sample.

[0025] Step 2.2: Based on the hard pseudo labels of the target domain samples described in step 2.1.2, the soft pseudo label distribution of the target domain samples is generated by using the Euclidean distance shown in formula (4) to calculate the similarity between the target domain samples, as shown in formula (5).

[0026]

[0027] Among them, λ is a hyperparameter used to balance the hard pseudo labels of samples and their similarity sets. Sample The most similar μ samples constitute a similar sample set

[0028] Corresponding similar tag sets

[0029] Step 3: Use the soft pseudo labels of the target domain samples trained by the model of unsupervised domain-adapted visible light and infrared single-modal pedestrian re-identification described in step 2 for unsupervised domain-adapted visible light and infrared cross-modal pedestrian re-identification training, thereby improving the generalization ability of unsupervised domain-adapted visible light and infrared cross-modal pedestrian re-identification.

[0030] Step 3.1: Use the soft pseudo labels of the target domain samples to extract class features for use in the cross-modal target domain sample feature mean, thereby obtaining the feature mean of the target domain as shown in formula (6), that is, the class prototype of the target domain.

[0031]

[0032] in, They represent the class prototypes of the kth class in the visible light modality and infrared modality in the target domain respectively.

[0033] Step 3.2: The class prototype of the target domain described in step 3.1 is trained with the target domain samples in the data distribution layer of the unsupervised domain-adapted visible light and infrared cross-modal pedestrian re-identification as shown in formula (7) to obtain the feature distribution of pedestrians with the same identity in the cross-modal neighbor feature space, thereby alleviating the modal difference.

[0034]

[0035] in, They represent the variance of visible light modality samples, infrared modality samples and their corresponding class prototypes respectively.

[0036] Step 3.3: The mitigated modality difference described in step 3.2 is optimized using the loss function shown in formula (8), thereby improving the generalization ability of unsupervised domain adaptation of visible light and infrared cross-modal pedestrian re-identification.

[0037]

[0038] in, It means that for each class label k∈{1,...,C}, the Euclidean distance of the corresponding visible light modality and infrared modality prototype is calculated. It represents the Euclidean distance of the variance in formula (7). It is used to minimize the modal differences of the same identity under different modalities, that is, to achieve cross-modal alignment effect, so as to improve generalization ability.

[0039] Beneficial Effects

[0040] Compared with the prior art, the method of the present invention overcomes the differences between domains by using camera driving strategies, hard false label allocation and soft false label distribution, and overcomes the modal differences between domains by reducing the distance between the feature means and variances of different modalities of the same sample, thereby enhancing the generalization ability of the unsupervised domain adapted visible light and infrared cross-modal pedestrian re-identification model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the principle of the present invention;

[0042] Figure 2 It is a schematic diagram of the process of the present invention;

[0043] Figure 3 This is the feature distance visualization result after using the method of the present invention;

[0044] Figure 4 This is the visualization result of feature distribution after using the method of the present invention. DETAILED DESCRIPTION

[0045] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the protection scope of the present invention.

[0046] In this embodiment, the Figure 1The principle shown in the figure is to send the source domain and the target domain into the network respectively. The source domain is the LLCM dataset, and the target domain is the SYSU-MM01 dataset. The input image is a visible light image and an infrared image with a size of 288×144 pixels.

[0047] Example

[0048] like Figure 2 As shown, the present embodiment is an unsupervised domain adaptation visible light and infrared cross-modal pedestrian re-identification method, and the specific implementation steps are as follows:

[0049] Step 1: Build a unimodal model for visible light and infrared unimodal person re-identification with unsupervised domain adaptation.

[0050] Step 1.1: Label the target domain dataset to be trained using a camera-driven method to form target domain training samples with camera labels.

[0051] Step 1.2: According to step 1.1, the target domain training samples and the source domain are sorted in a monotonically increasing order using the distribution difference shown in formula (9).

[0052]

[0053] Among them, D s , represents the source domain and the target domain, n∈{1,...,N} is the camera number, φ(·) represents the Gaussian kernel function, represents the reproducing Hilbert space, and |·| represents the total number of samples in the dataset.

[0054] In the embodiment, the training starts from the sub-target domain with the smallest distribution difference from the source domain, and is expanded every 5 epochs, and the next smallest sub-target domain is added in a monotonically increasing manner until it is expanded to the complete target domain.

[0055] Step 1.3: Classify the labeled samples in the source domain in visible light mode and infrared mode, and extract the features of the source domain samples and perform the calculation as shown in formula (10). Thus, the feature mean of the source domain is obtained, that is, the class prototype of the source domain.

[0056]

[0057] Among them, k∈{1,...,C} is the label of the class contained in the source domain, They represent the class prototypes of the kth class in the visible light modality and infrared modality in the source domain, Represent the number of samples of the kth class in the visible light modality and infrared modality respectively.

[0058] Step 2: Use the unsupervised domain adapted visible light and infrared single-modal pedestrian re-identification model described in step 1 to train the target domain samples and source domain class prototypes, so as to improve the generalization ability of unsupervised domain adapted visible light and infrared single-modal pedestrian re-identification.

[0059] Step 2.1: Construct the target domain sample feature pairs so that the target domain samples correspond to the source domain class prototypes in the form of labels.

[0060] Step 2.1.1: Get the label of the prototype of the source domain class with the smallest distance.

[0061] Step 2.1.2: Assign the label of the source domain class prototype in step 2.1.1 to the target domain sample to form a hard pseudo label of the target domain sample as shown in formula (11).

[0062]

[0063] in, Respectively represent the labels of the smallest source domain prototype visible light modality and infrared modality, denote the visible light modality samples and infrared modality samples of the target domain respectively, min(·) denotes the label of the minimum source domain class prototype, and the cosine distance between the visible light modality and infrared modality features of the target domain samples and the source domain class prototype is calculated to obtain the label of the source domain class prototype with the minimum distance and assign it to the target domain sample.

[0064] Step 2.2: Based on the hard pseudo labels of the target domain samples described in step 2.1.2, the soft pseudo label distribution of the target domain samples is generated by using the Euclidean distance shown in formula (12) to calculate the similarity between the target domain samples, as shown in formula (13).

[0065]

[0066] Among them, λ is a hyperparameter used to balance the hard pseudo labels of samples and their similarity sets. Sample The most similar μ samples constitute a similar sample set

[0067] Corresponding similar tag sets

[0068] In the embodiment, for samples in the target domain First, the index of the sample is used as the initial soft pseudo label, that is, Where m represents the modality to which the sample belongs (m=vis represents visible light modality, m=ir represents infrared modality). Since the indexes are independent of each other, this allocation method increases the feature distance between samples; in a batch, multiple images may correspond to the same pedestrian identity, and these image features are similar, so their distance should be smaller.

[0069] Step 3: Use the soft pseudo labels of the target domain samples trained by the model of unsupervised domain-adapted visible light and infrared single-modal pedestrian re-identification described in step 2 for unsupervised domain-adapted visible light and infrared cross-modal pedestrian re-identification training, thereby improving the generalization ability of unsupervised domain-adapted visible light and infrared cross-modal pedestrian re-identification.

[0070] Step 3.1: Use the soft pseudo labels of the target domain samples to extract class features for use in the cross-modal target domain sample feature mean, thereby obtaining the feature mean of the target domain as shown in formula (14), that is, the class prototype of the target domain.

[0071]

[0072] in, They represent the class prototypes of the kth class in the visible light modality and infrared modality in the target domain respectively.

[0073] Step 3.2: The class prototype of the target domain described in step 3.1 is trained with the target domain samples in the data distribution layer of the unsupervised domain-adapted visible light and infrared cross-modal pedestrian re-identification as shown in formula (15) to obtain the feature distribution of pedestrians with the same identity in the cross-modal neighbor feature space, thereby alleviating the modal difference.

[0074]

[0075] in, They represent the variance of visible light modality samples, infrared modality samples and their corresponding class prototypes respectively.

[0076] Step 3.3: The mitigated modality difference described in step 3.2 is optimized using the loss function shown in formula (16), thereby improving the generalization ability of unsupervised domain adaptation of visible light and infrared cross-modal pedestrian re-identification.

[0077]

[0078] in, It means that for each class label k∈{1,...,C}, the Euclidean distance of the corresponding visible light modality and infrared modality prototype is calculated. It represents the Euclidean distance of the variance in formula (7). It is used to minimize the modal differences of the same identity under different modalities, that is, to achieve cross-modal alignment effect, so as to improve generalization ability.

[0079] In the embodiment, the number of samples in a training batch is 64, and C=64 in formula (16).

[0080] To further demonstrate the superiority of the present invention, a visualization method is used to verify the generalization ability of unsupervised domain adaptation for visible light and infrared cross-modal pedestrian re-identification.

[0081] Figure 3 The feature distance visualization results after using the method of the present invention; the horizontal axis represents the distance between the query feature and the candidate feature, and the vertical axis represents the frequency of occurrence of the feature pair corresponding to the distance. Blue and green represent intra-class and inter-class distances, respectively. (a) represents the distance graph of the features of the SYSU-MM01 dataset extracted using the pre-training model, where the distribution of the intra-class distance and the inter-class distance almost overlap, indicating that the model does not have a good cross-modal pedestrian re-identification ability at this time; (b) represents the use of the trained model to extract features, and there is a clear distance between the distribution of the intra-class distance and the inter-class distance, indicating that the recognition ability of the model is better at this time.

[0082] Figure 4 This is the visualization result of feature distribution after using the method of the present invention; wherein, the circle and triangle represent visible light features and infrared features respectively, and different colors represent different pedestrian identities. (a) shows the distribution diagram of features extracted using the pre-trained model. The distribution of visible light modality and infrared modality features are relatively independent, and features of different identities are mixed together. (b) shows the distribution diagram of features extracted using the model trained with the algorithm proposed in this chapter. The feature distances of pedestrians of different identities are increased, the distribution of features of pedestrians of the same identity is more compact, and the features of visible light modality and infrared modality can be well clustered together, indicating that the algorithm proposed in this chapter has achieved significant results in extracting and distinguishing features of pedestrians of different modalities.

[0083] Table 1 Comparison of experimental results of visible light and infrared cross-modal pedestrian re-identification

[0084]

[0085] As can be seen from Table 1, the method of the present invention achieves the performance of Rank-1=45.1%, mAP=38.9%, mINP=22.0% for all scenes, and Rank-1=46.2%, mAP=53.7%, mINP=48.8% for indoor scenes under the unsupervised domain adaptation setting from LLCM to SYSU-MM01. In contrast, the current best supervised learning method DEEN can only achieve the performance of Rank-1=45.1%, mAP=38.9%, mINP=22.0% for all scenes, and Rank-1=10.9%, mAP=10.8%, mINP=4.4% for indoor scenes, and the current best unsupervised learning method PGM can only achieve the performance of Rank-1=9.5%, mAP=10.4%, mINP=4.8% for all scenes, and Rank-1=11.5%, mAP=19.2%, mINP=16.4% for indoor scenes. The above results show that the method of the present invention can significantly improve the generalization ability of the model.

[0086] The above description describes the implementation mode of the present invention in combination with the drawings and embodiments. However, for those skilled in the art, several improvements can be made without departing from the principle of this patent, and these also belong to the scope of protection of this patent.

Claims

1. A visible light and infrared cross-modal pedestrian re-identification method with unsupervised domain adaptation, characterized by: The following steps are included: Step 1: Build a unimodal model for unsupervised domain adaptation of visible light and infrared unimodal pedestrian re-identification; Step 2: Use the unsupervised domain adaptation model of visible light and infrared single-modal person re-identification described in step 1 to train the target domain samples and source domain class prototypes, so as to improve the generalization ability of unsupervised domain adaptation of visible light and infrared single-modal person re-identification; Step 3: Use the soft pseudo labels of the target domain samples trained by the model of unsupervised domain-adapted visible light and infrared single-modal pedestrian re-identification described in step 2 for unsupervised domain-adapted visible light and infrared cross-modal pedestrian re-identification training, thereby improving the generalization ability of unsupervised domain-adapted visible light and infrared cross-modal pedestrian re-identification.

2. The unsupervised domain adaptation visible light and infrared cross-modal pedestrian re-identification method according to claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: Label the target domain dataset to be trained by the training model using a camera-driven method to form target domain training samples with camera labels; Step 1.2: According to the target domain training samples described in step 1.1, the source domain training samples are sorted in a monotonically increasing order using the distribution difference shown in formula (1); Among them, D s , represents the source domain and the target domain, n∈{1,...,N} is the camera number, φ(·) represents the Gaussian kernel function, represents the regenerated Hilbert space, |·| represents the total number of samples in the dataset; Step 1.3: Classify the labeled samples of the source domain in visible light mode and infrared mode, and extract features from the source domain samples and perform calculations as shown in formula (2); thereby obtaining the feature mean of the source domain, i.e., the class prototype of the source domain; Among them, k∈{1,...,C} is the label of the class contained in the source domain, They represent the class prototypes of the kth class in the visible light modality and infrared modality in the source domain, Represent the number of samples of the kth class in the visible light modality and infrared modality respectively.

3. The unsupervised domain adaptation visible light and infrared cross-modal pedestrian re-identification method according to claim 1, characterized in that: Step 2 is implemented as follows: Step 2.1: Construct the target domain sample feature pairs so that the target domain samples correspond to the source domain class prototypes in the form of labels; Step 2.1.1: Get the label of the prototype of the source domain class with the smallest distance; Step 2.1.2: Assign the label of the source domain class prototype in step 2.1.1 to the target domain sample to form a hard pseudo label of the target domain sample as shown in formula (3); in, Respectively represent the labels of the smallest source domain prototype visible light modality and infrared modality, denote the visible light modality samples and infrared modality samples of the target domain respectively, min(·) denotes the label of the minimum source domain class prototype, and the cosine distance between the visible light modality and infrared modality features of the target domain samples and the source domain class prototype is calculated to obtain the label of the source domain class prototype with the minimum distance and assign it to the target domain sample; Step 2.2: Based on the hard false labels of the target domain samples described in step 2.1.2, the soft false label distribution of the target domain samples is generated by using the Euclidean distance shown in formula (4) for the similarity between the target domain samples, as shown in formula (5); Among them, λ is a hyperparameter used to balance the hard pseudo labels of samples and their similarity sets. Sample The most similar μ samples constitute a similar sample set Corresponding similar tag sets 4. The unsupervised domain adaptation visible light and infrared cross-modal pedestrian re-identification method according to claim 1, characterized in that: Step 3 is implemented as follows: Step 3.1: Use the soft pseudo labels of the target domain samples to extract class features for use in the cross-modal target domain sample feature mean, thereby obtaining the feature mean of the target domain as shown in formula (6), that is, the class prototype of the target domain; in, They represent the class prototypes of the kth class in the visible light modality and infrared modality in the target domain respectively; Step 3.2: The class prototype of the target domain described in step 3.1 is trained with the target domain samples in the data distribution layer of the visible light and infrared cross-modal pedestrian re-identification of unsupervised domain adaptation as shown in formula (7) to obtain the feature distribution of pedestrians with the same identity in the cross-modal neighbor feature space, thereby alleviating the modality difference; in, Respectively represent the variance of visible light modality samples, infrared modality samples and their corresponding class prototypes; Step 3.3: The mitigation modality difference described in step 3.2 is optimized using the loss function shown in formula (8), thereby improving the generalization ability of unsupervised domain adaptation of visible light and infrared cross-modal pedestrian re-identification; in, It means that for each class label k∈{1,...,C}, the Euclidean distance of the corresponding visible light modality and infrared modality prototype is calculated. represents the Euclidean distance of the variance in formula (7); It is used to minimize the modal differences of the same identity under different modalities, that is, to achieve cross-modal alignment effect, so as to improve generalization ability.

5. A computer-readable storage device storing a computer program, characterized in that: When the computer program is executed, the visible light and infrared cross-modal pedestrian re-identification method with unsupervised domain adaptation as claimed in any one of claims 1 to 4 is implemented.

6. A visible light and infrared cross-modal pedestrian re-identification system with unsupervised domain adaptation, comprising a storage device, a processor, and a computer program stored in the storage device and executable on the processor, characterized in that: The processor executes the computer program to implement the visible light and infrared cross-modal pedestrian re-identification method with unsupervised domain adaptation as claimed in any one of claims 1 to 4.

Images