An Unsupervised Cross-Modal Person Re-Identification Method

Abstract

An unsupervised cross-modal pedestrian re-identification method solves the problem that existing cross-modal pedestrian re-identification methods rely on labeled data, and belongs to the field of pedestrian re-identification technology. The present invention includes: combining three single-channel pedestrian visible light data sets with corresponding pedestrian images in an infrared data set; using a momentum encoder M to extract features and cluster the combined data, determine the training sample set of each cluster, and determine the pseudo label and the center of mass of each cluster; calculating the intersection-over-union ratio of the images in each cluster, and using the obtained intersection-over-union ratio as the element in the cluster consistency matrix. The intersection of the clusters corresponding to the elements in the matrix that reach the threshold is used as the refinement result, and using the softmax loss function, the difficult triplet loss function, and the modal contrast loss function to optimize E, and updating M based on the optimized E; using the updated M to identify the three-channel visible light and infrared pedestrian images in the test data set.

Description

Technical Field

[0001] The present invention relates to an unsupervised cross-modal pedestrian re-identification method, belonging to the technical field of pedestrian re-identification. Background Art

[0002] Person re-identification (PRED) aims to find images of a person with a given identity from a large-scale image library. With the introduction of deep learning techniques, early PRED methods can achieve good performance within the visible light modality. However, these methods can only be used under conditions with sufficient lighting. Currently, most surveillance cameras automatically switch from visible light mode to infrared mode when lighting is insufficient. Therefore, cross-modal PRED between visible light and infrared has attracted increasing attention. The goal of cross-modal PRED is to match people with the same identity between the visible and infrared modalities.

[0003] While existing supervised learning-based cross-modal person re-identification methods have achieved promising performance, they typically rely on large-scale cross-modal annotated datasets. It is well known that annotating datasets for image retrieval tasks is a time-consuming task, and the differences between modalities further increase the difficulty of annotation. To address these issues, researchers have proposed the first unsupervised cross-modal person re-identification method, H2H. This method first pre-trains the model on a labeled source domain dataset (a unimodal dataset) and then performs homogeneous-heterogeneous learning on an unlabeled target domain dataset (a cross-modal dataset). Although H2H no longer relies on identity information in cross-modal scenarios, it still requires identity information from other domains for pre-training. Therefore, H2H is not a completely unsupervised method, but rather a cross-domain method. Cross-domain methods not only require additional data pre-processing operations but also have certain requirements for the source domain data. In addition, suitable source domains are not always available. Summary of the Invention

[0004] To address the problem that existing cross-modal person re-identification methods rely on labeled data, the present invention provides an unsupervised cross-modal person re-identification method.

[0005] The present invention provides an unsupervised cross-modal person re-identification method, comprising:

[0006] S1. Collect multiple pedestrian images in visible light and infrared modes respectively, and construct visible light dataset and infrared dataset;

[0007] S2. Extract red, green, and blue single-channel data from each visible light pedestrian image in the visible light dataset, and form three single-channel visible light datasets respectively. The three single-channel visible light datasets are then combined with the infrared dataset to obtain three new single-channel datasets.

[0008] S3. Use the momentum encoder M to extract features from the three single-channel new data sets, obtain three single-channel feature sets respectively, and then perform clustering based on the three single-channel feature sets, assign a pseudo label to each training sample, and calculate the centroid of each cluster based on the pseudo label;

[0009] S4. Calculate the intersection-and-union ratio of the images in each cluster in the three single-channel new data sets, and use the obtained intersection-and-union ratio as an element in the cluster consistency matrix U;

[0010] S5. The intersection of the clusters corresponding to the elements reaching the threshold in the cluster consistency matrix U is taken as the refined result, and all channels of the same image in the intersection are added to the refined result;

[0011] S6. Optimize the online encoder E using a softmax loss function, a difficult triplet loss function, and a modality contrast loss function according to the obtained centroid and refinement results, and then update the momentum encoder M based on the optimized online encoder E according to a weighted method;

[0012] S7. Use the updated momentum encoder M to extract features from the three channels of the visible light image and the infrared image in the test data set, and obtain a pedestrian image with a specific identity based on the feature distance.

[0013] Preferably, in S3, DBSCAN is used to cluster the feature sets of the three single channels respectively, and a pseudo label is assigned to each training sample.

[0014] As a preference, in S3, the centroid c of each cluster p :

[0015]

[0016] Among them, n p is the total number of samples in the corresponding cluster, m l The features of the training samples in the corresponding clusters are extracted from the three single-channel training samples.

[0017] As a preference, in S4, the intersection-over-union ratio is used as the cluster consistency matrix Any element u i,j,k for:

[0018]

[0019] Where |·| is the function for calculating the number of samples in the set. In the process of calculating the intersection-over-union ratio, different channels of the same image are regarded as the same sample; n r 、n g and n b Represents the number of clusters in three single channels, Indicates {Xr ,X ir The set of samples in the i-th cluster in}, i = 1, 2…n r , Indicates {X g ,X ir The set of samples in the jth cluster in}, j = 1, 2, ..., n g , Indicates {X b ,X ir The sample set in the kth cluster in}, k=1,2,…,n b .

[0020] Preferably, in S6, the online encoder E is optimized using the softmax loss function:

[0021]

[0022] Among them, L soft represents the softmax loss function, f p is any sample x extracted by the online encoder E from the refined result p Characteristics, c p is x p The centroid of the positive cluster, n c is the number of clusters, τ s is the temperature hyperparameter of the softmax loss, c a is the centroid of any cluster, a=1,2,…,n c , and the refined result and any sample x p The centroid of the cluster with the same pseudo label is called x p The centroid of the positive cluster is called x p The negative cluster centroid of .

[0023] Preferably, in S6, the online encoder E is optimized using the difficult triplet loss function:

[0024]

[0025] Among them, L hard represents the difficult triple loss function, mar is the threshold hyperparameter, dist(·,·) is the Euclidean distance metric function, and the online encoder E selects a minibatch of size P×K from the refined results. is an arbitrarily selected sample in the minibatch of size P×K, for The most difficult positive sample, for The hardest negative example.

[0026] Preferably, in S6, the online encoder E is optimized using the modality contrast loss function:

[0027]

[0028] Among them, L moda represents the modality contrast loss function, x pq represents the sample with modality label q in the p-th cluster extracted by the online encoder E in the refined result, q∈{0,1} is the modality label, q=0 represents visible modality, q=1 represents infrared modality, f pq is x pq Characteristics, c pl is x pq The positive mode centroid, subscript l = 1-q, τ m is the temperature hyperparameter of the mode contrast loss, c sd Indicates that x pq The centroid of the modalities with different cluster labels, subscript s represents clusters other than p, subscript d∈{0,1}, d=0 represents visible modality, d=1 represents infrared modality, Q represents x pq The set of the most difficult negative modal centroids.

[0029] Preferably, the modal centroid of the modal label q in the p-th cluster is:

[0030]

[0031] m h is the sample feature of the modality label q in cluster p extracted by the online encoder E, n pq represents the total number of samples with modality label q in cluster p.

[0032] The beneficial effects of the present invention are that the present invention no longer relies on any source domain data for learning, thereby improving the flexibility and scalability of the recognition model; using different channels of the same sample as positive samples can effectively improve the robustness of the model to brightness and contrast; and comparing the clustering results of the three channels in the visible modality can improve the reliability of the labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the unsupervised cross-modal person re-identification method of the present invention;

[0034] Figure 2 It is a schematic diagram of the principle of the present invention;

[0035] Figure 3 Schematic diagram of the principle of label refining of the present invention;

[0036] Figure 4 is the modal contrast loss optimization graph of the present invention; DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0038] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0039] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0040] The unsupervised cross-modal person re-identification method of this embodiment includes:

[0041] Step 1: Collect multiple pedestrian images in visible light and infrared modes respectively, and construct visible light dataset and infrared dataset;

[0042] Step 2: Extract red, green, and blue single-channel data from each visible light pedestrian image in the visible light dataset, and form three single-channel visible light datasets respectively. The three single-channel visible light datasets are then combined with the infrared dataset to obtain three single-channel training samples.

[0043] Specifically, in step 2 of this embodiment, the visible light data set Extract different channels and get the red channel respectively Green Channel and blue channel Combine the three channel visible light dataset with the infrared modality Combine to get three new single-channel datasets {X r ,X ir}, {X g ,X ir} and {X b ,X ir}.

[0044] Step 3: Use the momentum encoder M to extract features from the three single-channel new data sets, obtain three single-channel feature sets, and then perform clustering based on the three single-channel feature sets, assign a pseudo label to each training sample, and calculate the centroid of each cluster based on the pseudo label;

[0045] Specifically, in step 3 of this embodiment, the momentum encoder M is used to train the three single-channel training sample sets {X r ,X ir}, {X g ,Xir} and {X b ,X ir} Perform feature extraction separately; then use DBSCAN (Density-Based Spatial Clustering of Applications with Noise) to cluster the three feature sets obtained and assign pseudo labels to each training sample;

[0046] Calculate the centroid c of each cluster p :

[0047]

[0048] Among them, n p is the total number of samples in the corresponding cluster, m l The features of the training samples in the corresponding clusters extracted from the three single-channel training samples;

[0049] Step 4: Calculate the intersection-and-union ratio of the number of images in the training sample set of each cluster in the three single channels, and use the obtained intersection-and-union ratio as an element in the cluster consistency matrix U;

[0050] Cluster consistency matrix of this embodiment Any element u i,j,k for:

[0051]

[0052] Where |·| is the function for calculating the number of samples in the set. In the process of calculating the intersection-over-union ratio, different channels of the same image are regarded as the same sample; n r 、n g and n b Represents the number of clusters in three single channels, Indicates {X r ,X ir The set of samples in the i-th cluster in}, i = 1, 2…n r , Indicates {X g ,X ir The set of samples in the jth cluster in}, j = 1, 2, ..., n g , Indicates {X b ,X ir The sample set in the kth cluster in}, k=1,2,…,n b .

[0053] In the process of calculating the intersection-over-union ratio, different channels of the same image are regarded as the same sample. Figure 3For example, the superscripts represent the channel and mode, and the subscripts represent the sample number. and From the same sample x2, the training samples of a cluster in the red channel include: The training samples of a cluster in the green channel include: The training samples of a cluster in the blue channel include: The sample images of the intersection of these three clusters are: x2, x3, x6, x7, x8, and the sample images of the union of these three clusters are: x6,x7,x8,x9,x 10 , the intersection-and-union ratio of the number is As the cluster consistency matrix An element in

[0054] Step 5: The intersection of the clusters corresponding to the elements that reach the threshold in the cluster consistency matrix U is taken as the refined result, and all channels of the same image in the intersection are added to the refined result;

[0055] by Figure 3 For example, the threshold is t, If it is greater than t, all channels of the image in the intersection x2, x3, x6, x7, x8 are taken as the refined result, that is, the refined result is:

[0056] It should be noted that the advantages of doing so are twofold: on the one hand, although different channels of the same sample differ in brightness and contrast, they are essentially positive samples of each other. By optimizing the distance between them, the model's robustness to brightness and contrast can be improved; on the other hand, refinement will reduce the amount of training data, and compared with infrared images, the number of visible images is reduced more. Adding images from multiple channels can supplement the training set.

[0057] Step 6: Optimize the online encoder E using the softmax loss function, the difficult triplet loss function, and the modality contrast loss function according to the obtained centroid and refinement results, and then update the momentum encoder M based on the optimized online encoder E according to the weighted method;

[0058] Specifically, the online encoder E is optimized using the softmax loss function, which aims to increase x p Similarity to the centroid of the positive cluster and reduce x p Similarity to the negative cluster centroid:

[0059]

[0060] L soft represents the softmax loss function, f pis any sample x extracted by the online encoder E from the refined result p Characteristics, c p is x p The centroid of the positive cluster, n c is the number of clusters, τ s is the temperature hyperparameter of the softmax loss, c a is the centroid of any cluster, a=1,2,…,n c , and the refined result and any sample x p The centroid of the cluster with the same pseudo label is called x p The centroid of the positive cluster is called x p The negative cluster centroid of .

[0061] Specifically, the online encoder E selects a minibatch of size P×K from the refined results, and the difficult triplet loss function first selects a sample As an anchor point, then add With the most difficult positive sample The similarity of With the most difficult negative sample Similarity:

[0062]

[0063] Where mar is the threshold hyperparameter and dist(·,·) is the Euclidean distance metric function.

[0064] This implementation requires finding the Positive and negative samples, and then based on the sample The most difficult thing to determine is the distance relationship. The closest negative sample is defined as the most difficult negative sample, The positive sample with the farthest distance is defined as the most difficult positive sample.

[0065] The modal centroid is calculated based on the clustering results. The modal centroid c of the modal label q in the p-th cluster is pq is defined as:

[0066]

[0067] m h is the sample feature of the modality label q in cluster p extracted by the online encoder E, n pq represents the total number of samples with modality label q in cluster p.

[0068] will be compared with the sample x pq The modality centroid c with the same cluster label and different modality labels pl (l=1-q) is called xpq The positive modal centroid of q∈{0,1} is the modal label, q=0 represents the visible modality, q=1 represents the infrared modality, and will be compared with the sample x pq Modality centroid c with different cluster labels sd (s≠p) is called x pq The negative modal centroid of , the subscript s is the cluster other than p, the subscript d∈{0,1}, d=0 represents the visible mode, d=1 represents the infrared mode. Figure 4 As shown, for any labeled sample x pq , the purpose of the modality contrast loss is to increase its similarity with the positive modality centroid and decrease its similarity with the negative modality centroid:

[0069]

[0070] x pq represents the sample with modality label q in the p-th cluster extracted by the online encoder E in the refined result, f pq is x pq Characteristics, c pl is x pq The positive mode centroid, subscript l = 1-q, τ m is the temperature hyperparameter of the mode contrast loss, c sd Indicates that x pq The modal centroid with different cluster labels, Q represents x pq The set of the most difficult negative modal centroids.

[0071] Step 7: Use the updated momentum encoder M to extract features from the three channels of the visible light image and the infrared image in the test data set, and obtain pedestrian images with specific identities based on feature distances.

[0072] That is to say, the optimized momentum encoder is used to extract features from the three channels of the visible light image and the infrared image in the test data set, and then the pedestrian images with specific identities are found based on the similarity between the features.

[0073] This implementation no longer relies on any source domain data for learning, which improves the flexibility and scalability of the recognition model; using different channels of the same sample as positive samples can effectively improve the model's robustness to brightness and contrast; and comparing the clustering results of the three channels in the visible modality can improve the reliability of the labels.

[0074] The unsupervised cross-modal person re-identification method based on label refinement proposed in the embodiment of the present invention is further illustrated below through a specific embodiment.

[0075] Specific embodiment: This embodiment uses mean average precision (mAP), rank-1 and rank-10 to evaluate the performance of the DHCR of the present invention and compare it with the existing fully unsupervised methods, among which the selected existing fully unsupervised methods are SpCL, ICE, CIFL and H2H.

[0076] The experimental datasets used include: SYSU-MM01 and RegDB, among which,

[0077] The SYSU-MM01 dataset contains 30,071 visible light images and 15,792 infrared images of 491 identities captured by 6 cameras (4 infrared and 2 visible). The training set contains 395 identities and the test set contains 96 identities.

[0078] The RegDB dataset contains 8,240 images from 412 identities, each of which contains 10 visible light images and 10 infrared images. 206 identities are used as training sets and 206 identities are used as test sets.

[0079] Table 1 shows the comparison results of the present invention with the existing unsupervised person re-identification methods on the SYSU-MM01 and RegDB datasets.

[0080]

[0081] It can be seen that compared with the optimal method H2H, the present invention has achieved obvious advantages on both SYSU-MM01 and RegDB. Therefore, the comprehensive performance of the present invention is better than the existing unsupervised pedestrian re-identification method.

[0082] In summary, the unsupervised cross-modal person re-identification method based on label refinement proposed in an embodiment of the present invention can not only prevent the misalignment of identities when aligning the feature distributions of the two modalities, but also promote the positive samples from different modalities to be clustered into the same cluster, thereby providing sufficient positive sample pairs between modalities for optimization; in addition, the reliability of pseudo-labels can be improved by comparing the clustering results of the three channels in the visible modality.

[0083] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be employed in conjunction with other described embodiments.

Claims

1. An unsupervised cross-modal person re-identification method, characterized by: The method comprises: S1. Collect multiple pedestrian images in visible light and infrared modes respectively, and construct visible light dataset and infrared dataset; S2. Extract red, green, and blue single-channel data from each visible light pedestrian image in the visible light dataset, and form three single-channel visible light datasets respectively. The three single-channel visible light datasets are then combined with the infrared dataset to obtain three new single-channel datasets. S3. Use the momentum encoder M to extract features from the three single-channel new data sets, obtain three single-channel feature sets respectively, and then perform clustering based on the three single-channel feature sets, assign a pseudo label to each training sample, and calculate the centroid of each cluster based on the pseudo label; S4, calculate the intersection-and-union ratio of the images in each cluster in the three single-channel new data sets, and use the obtained intersection-and-union ratio as the element in the cluster consistency matrix U; In S4, the intersection-and-union ratio is used as the element in the cluster consistency matrix Any element u i,j,k for: Where |·| is the function for calculating the number of samples in the set. In the process of calculating the intersection-over-union ratio, different channels of the same image are regarded as the same sample; n r 、n g and n b Represents the number of clusters in three single channels, Indicates {X r ,X ir The set of samples in the i-th cluster in}, i = 1, 2…n r , Indicates {X g ,X ir The sample set in the jth cluster in}, j=1,2,…,n g , Indicates {X b ,X ir The sample set in the kth cluster in}, k=1,2,…,n b ; S5. The intersection of the clusters corresponding to the elements reaching the threshold in the cluster consistency matrix U is taken as the refined result, and all channels of the same image in the intersection are added to the refined result; S6. Optimize the online encoder E using the softmax loss function, the difficult triplet loss function, and the modality contrast loss function based on the obtained centroid and refinement results, and then update the momentum encoder M based on the optimized online encoder E according to the weighted method; In S6, the modality contrast loss function is used to optimize the online encoder E: Among them, L moda represents the modality contrast loss function, x pq represents the sample with modality label q in the p-th cluster extracted by the online encoder E in the refined result, q∈{0,1} is the modality label, q=0 represents visible modality, q=1 represents infrared modality, f pq is x pq Characteristics, c pl is x pq The positive mode centroid, subscript l = 1-q, τ m is the temperature hyperparameter of the mode contrast loss, c sd Indicates that x pq The centroid of the modalities with different cluster labels, subscript s represents clusters other than p, subscript d∈{0,1}, d=0 represents visible modality, d=1 represents infrared modality, Q represents x pq The set of the most difficult negative modal centroids; S7. Use the updated momentum encoder M to extract features from the three channels of the visible light image and the infrared image in the test data set, and obtain a pedestrian image with a specific identity based on the feature distance.

2. The unsupervised cross-modal person re-identification method according to claim 1, characterized in that In S3, DBSCAN is used to cluster the three single-channel feature sets separately and assign pseudo labels to each training sample.

3. The unsupervised cross-modal person re-identification method according to claim 2, characterized in that: In S3, the centroid c of each cluster p : Among them, n p is the total number of samples in the corresponding cluster, m l The features of the training samples in the corresponding clusters are extracted from the three single-channel training samples.

4. The unsupervised cross-modal person re-identification method according to claim 1, characterized in that In S6, the softmax loss function is used to optimize the online encoder E: Among them, L soft represents the softmax loss function, f p is any sample x extracted by the online encoder E from the refined result p Characteristics, c p is x p The centroid of the positive cluster, n c is the number of clusters, τ s is the temperature hyperparameter of the softmax loss, c a is the centroid of any cluster, a=1,2,…,n c , and the refined result and any sample x p The centroid of the cluster with the same pseudo label is called x p The centroid of the positive cluster is called x p The negative cluster centroid of .

5. The unsupervised cross-modal person re-identification method according to claim 1, characterized in that In S6, the online encoder E is optimized using the difficult triplet loss function: Among them, L hard represents the difficult triple loss function, mar is the threshold hyperparameter, dist(·,·) is the Euclidean distance metric function, and the online encoder E selects a minibatch of size P×K from the refined results. is an arbitrarily selected sample in the minibatch of size P×K, for The most difficult positive sample, for The hardest negative example.

6. The unsupervised cross-modal person re-identification method according to claim 1, characterized in that The modal centroid of the modal label q in the p-th cluster is: m h is the sample feature of the modality label q in cluster p extracted by the online encoder E, n pq represents the total number of samples with modality label q in cluster p.

7. A computer-readable storage device storing a computer program, characterized in that: When the computer program is executed, the unsupervised cross-modal person re-identification method according to any one of claims 1 to 6 is implemented.

8. An unsupervised cross-modal person re-identification system, comprising a storage device, a processor, and a computer program stored in the storage device and executable on the processor, characterized in that: The processor executes the computer program to implement the unsupervised cross-modal person re-identification method according to any one of claims 1 to 6.

Images