A method for visible light and infrared cross-modal pedestrian re-identification based on intermediate domain

Abstract

The application discloses a kind of based on intermediate field's visible light and infrared cross-modal pedestrian re-identification method, belong to computer vision technical field.The application includes the following steps: 1, the feature of intermediate field is generated using the weight of source domain and target domain in intermediate field generator;2, the distribution difference of intermediate field and source domain, target domain is optimized using cross-entropy loss, and the distance between fields is optimized using L2 norm;3, the feature of visible light and infrared in the optimized target domain is assigned pseudo-label p using DBSCAN clustering method, and the feature is stored using feature storage dictionary;4, the graph representation of clustering under visible light modality and infrared modality is constructed to calculate the similarity of node;5, the similarity of visible light modality and infrared modality is weighted bipartite graph clustering feature matching;6, the contrast learning of clustering feature matching is carried out using ClusterNCE loss.Compared with prior art, the application can improve the generalization ability in visible light and infrared cross-modal pedestrian re-identification method.

Description

TECHNICAL FIELD

[0001] The application relates to a visible light and infrared cross-modal pedestrian re-identification method based on an intermediate domain and belongs to the technical field of computer vision. BACKGROUND

[0002] The visible light and infrared cross-modal pedestrian re-identification task is an important challenge in the field of computer vision and has important significance for all-weather video monitoring, intelligent security and people flow analysis and the like.

[0003] Although the visible light and infrared cross-modal pedestrian re-identification method based on supervised learning has made certain progress, the application range of these methods is limited due to the high cost of obtaining high-quality labeled data. In contrast, unsupervised learning without labels is more practical in the visible light and infrared cross-modal pedestrian re-identification.

[0004] However, most of the current researches focus on the performance of the model on a specific data set, ignoring the research on the generalization ability of the model, resulting in a significant decline in the performance of the algorithm when the data domain changes, thereby limiting the practicality.

[0005] Therefore, how to improve the generalization ability in the visible light and infrared cross-modal pedestrian re-identification method when the data domain changes has become a problem to be solved. SUMMARY

[0006] The application aims to solve the technical problem of low generalization ability in the existing visible light and infrared cross-modal pedestrian re-identification method. A visible light and infrared cross-modal pedestrian re-identification method based on an intermediate domain is proposed.

[0007] The application aims to achieve the following technical solutions:

[0008] The application provides a visible light and infrared cross-modal pedestrian re-identification method based on an intermediate domain, which comprises the following steps:

[0009] Step 1: generating the features of the intermediate domain by using the source domain and target domain weights in the intermediate domain generator;

[0010] Step 1.1: in the intermediate domain generator, the weights of the source domain and the target domain are obtained in the manner shown in formula (1);

[0011]

[0012] wherein, w s , w t respectively represent the weights of the source domain and the target domain in generating the intermediate domain, MLP represents a multi-layer perception machine, and FC represents a fully connected layer. avg , F gem , Fmax respectively represent the features after average pooling, generalized average pooling, and max pooling. δ(·) represents a softmax function.

[0013] Step 1.2: Use the weights of the source domain and the target domain described in step 1.1 in the intermediate domain to form the features of the intermediate domain;

[0014] F m = w s F s +w t F t (2)

[0015] where w s represents the source domain, w t represents the target domain, F s represents the source domain weight, F t represents the target domain weight. F m is the feature of the intermediate domain;

[0016] Step 2: Use cross-entropy loss to optimize the distribution difference between the intermediate domain and the source domain and the target domain, and use L2 norm to optimize the distance between the domains;

[0017] Step 2.1: Use cross-entropy loss to optimize the distribution difference between the intermediate domain and the source domain and the target domain in the manner shown in equation (3);

[0018]

[0019] where k is the domain identifier, indicating that the feature belongs to the source domain or the target domain, i is the index in a small batch, and φ is the classifier. represents the loss function, and a small value indicates good optimization effect.

[0020] Step 2.2: Use L2 norm to optimize the distance between the domains in the feature space in the manner shown in equation (4);

[0021]

[0022] where, represents the minimization of the distance between the source domain and the intermediate domain, and the target domain and the intermediate domain.

[0023] Step 3: Use DBSCAN clustering method to assign pseudo labels p to the visible light and infrared features in the target domain optimized in step 2, and use the feature storage dictionary in the manner shown in equation (5) to store the features.

[0024]

[0025] where f(·) represents a feature extractor, Vp , I p , VI p represents the feature set with identity p in the visible light modality, the infrared modality, and the visible light and infrared mixed modality, is the feature set with identity p in the visible light modality; is the feature set with identity p in the infrared modality; is the feature set with identity p in the visible light and infrared mixed modality; is the feature storage dictionary in the visible light modality; is the feature storage dictionary in the infrared modality; is the feature storage dictionary in the visible light and infrared mixed modality;

[0026] Step 4: Constructing the graph representation of the clustering in the visible light modality and the infrared modality for calculating the similarity between nodes;

[0027] Step 4.1: Constructing the graph representation of the clustering in the visible light modality and the infrared modality in the manner shown in formula (6);

[0028]

[0029] wherein the graph representations of the visible light modality and the infrared modality are [I], Y vis , Y ir represents the number of clusters.

[0030] Step 4.2: Calculating the similarity of the visible light modality and the infrared modality in the manner shown in formula (7) for the graph representation described in step 4.1;

[0031]

[0032] wherein, respectively represents the cluster the number of instances in the cluster the feature of the cluster is represented by the feature mean of the instances, represents the feature mean of the visible light modality with label i; represents the feature mean of the infrared modality with label i;

[0033] Step 5: Performing weighted bipartite graph clustering feature matching for the similarity of the visible light modality and the infrared modality described in step 4.

[0034]

[0035] wherein, is an identifier for representing node and node whether they belong to the same identity. (i, j) represents the index of the matching relationship to be extended in the matching matrix Q; Q a，b = 1, the corresponding feature vector; d(·) is the Euclidean distance; m represents the matching direction, which is used to ensure that each node is matched. m = vis represents the matching from the visible light modality to the infrared modality, and m = ir represents the matching from the infrared modality to the visible light modality;

[0036] Step 6: The cluster feature matching in step 5 is subjected to contrast learning in a manner as shown in formula (9) by using the ClusterNCE loss;

[0037]

[0038]

[0039] wherein, represents the stored dictionary visible light pedestrian feature, the stored dictionary infrared pedestrian feature, and τ is a hyperparameter used to control the distance between classes. F vis is the feature of visible light, ir is the feature of infrared, and [F vis , F ir ] is the mixed feature;

[0040] Advantages:

[0041] 1. For the problem of inter-domain difference, the intermediate domain is generated by mixing the hidden representations of the source domain and the target domain features, so that the model learns more consistent feature representations, thereby reducing the domain difference. For the intra-domain modality difference problem, the supervised learning method is used to optimize the labeled source domain. For the unlabeled target domain, on the basis of clustering different modalities, clustering of mixed modalities is added to indirectly establish the cross-modality correspondence. The weighted bipartite graph matching is used to further align the visible light modality and the infrared modality, thereby reducing the modality difference.

[0042] 2. Compared with the prior art, the method of the present application can maintain a high recognition accuracy when the data domain changes, and has practicality. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a schematic diagram of the principle of the present application;

[0044] Figure 2 is the feature distance visualization result after using the method of the present application;

[0045] Figure 3 is the feature distribution visualization result after using the method of the present application;

[0046] Figure 4 The visible light and infrared retrieval results are those obtained by the method of this invention.

[0047] Figure 5 These are the infrared and visible light retrieval results of the method of this invention. Detailed Implementation

[0048] To better illustrate the purpose and advantages of this invention, the invention will be further described below with reference to the accompanying drawings and examples. It should be noted that the implementation of this invention is not limited to the following embodiments, and any modifications or alterations made to this invention will fall within the scope of protection of this invention.

[0049] Example

[0050] like Figure 1 As shown, the specific implementation steps of the visible light and infrared cross-modal pedestrian re-identification method based on the intermediate domain of the present invention are as follows:

[0051] First, the source and target domains are fed into the network. In this embodiment, the source domain is the SYSU-MM01 dataset, and the target domain is the LLCM dataset. The input images are visible light images and infrared images, with a size of 288×144 pixels.

[0052] Step 1: Generate features of the intermediate domain using the weights of the source and target domains in the intermediate domain generator;

[0053] Step 1.1: In the intermediate domain generator, the weights of the source domain and the target domain are obtained in the manner shown in Equation (1);

[0054]

[0055] Among them, w s w t These represent the weights of the source and target domains in generating the intermediate domain, respectively. MLP stands for Multilayer Perceptron, and FC stands for Fully Connected Layer. avg F gem F max These represent the characteristics after average pooling, generalized average pooling, and max pooling, respectively. δ(·) represents the softmax function.

[0056] Step 1.2: Apply the weights of the source domain and target domain described in Step 1.1 to the intermediate domain to form the features of the intermediate domain;

[0057] F m =w s F a +w t F t (2)

[0058] Among them, w sdenotes the source domain, w t denotes the target domain, F s denotes the source domain weight, F t denotes the target domain weight. F m is the feature of the intermediate domain;

[0059] Step 2: the distribution difference between the intermediate domain and the source domain and the target domain is optimized by using cross-entropy loss, and the distance between the domains is optimized by using L2 norm;

[0060] Step 2.1: the distribution difference between the intermediate domain and the source domain and the target domain is optimized in the manner shown in formula (3) by using cross-entropy loss;

[0061]

[0062] wherein k is a domain identifier, indicating that the feature belongs to the source domain or the target domain, i is an index in a small batch, and φ is a classifier. denotes the loss function, and a small value indicates good optimization effect.

[0063] Step 2.2: the distance between the domains in the feature space is optimized in the manner shown in formula (4) by using L2 norm;

[0064]

[0065] wherein, denotes the minimization of the distance between the source domain and the intermediate domain, and the target domain and the intermediate domain.

[0066] Step 3: the DBSCAN clustering method is used to assign pseudo-labels p to the visible light and infrared features in the target domain optimized in step 2, and the features are stored in the manner shown in formula (5) by using the feature storage dictionary.

[0067]

[0068]

[0069] wherein f(·) denotes a feature extractor, V p , I p , VI p denotes the feature set of identity p in the visible light modality, the infrared modality, and the visible light and infrared mixed modality, is the feature set of identity p in the visible light modality; is the feature set of identity p in the infrared modality; is the feature set of identity p in the visible light and infrared mixed modality; is the feature storage dictionary in the visible light modality; is the feature storage dictionary in the infrared modality; to store the dictionary of features under visible light and infrared mixed modalities;

[0070] In the embodiment, the maximum distance of clustering of DBSCAN is set to 0.6, and the minimum number of instances contained in each cluster is set to 4.

[0071] Step 4: Constructing the graph representation of clustering under visible light modalities and infrared modalities for calculating the similarity between nodes;

[0072] Step 4.1: Constructing the graph representation of clustering under visible light modalities and infrared modalities in the manner shown in formula (6);

[0073]

[0074] wherein the graph representations of visible light modalities and infrared modalities are respectively [I],Y vis , Y ir represents the number of clusters.

[0075] Step 4.2: Calculating the similarity of visible light modalities and infrared modalities in the manner shown in formula (7) for the graph representation described in step 4.1;

[0076]

[0077] wherein, respectively represent the number of instances in the cluster cluster The number of instances in the cluster is represented by the feature mean of the instances, represents the feature mean of visible light modalities with label i; represents the feature mean of infrared modalities with label i;

[0078] Step 5: Performing weighted bipartite graph clustering feature matching for the similarity of visible light modalities and infrared modalities described in step 4.

[0079]

[0080]

[0081] wherein, is an identifier for representing whether nodes and nodes belong to the same identity. (i, j) represents the matching relationship to be expanded in the index of the matching matrix Q; represents Q a，bcorresponds to the feature vector when Q

[0082] In the embodiment, Q m = 1 indicates the same identity, and Q m = 0 indicates different identities. m indicates the matching direction, and is used to ensure that each node is matched.

[0083] Step 6: The cluster feature matching in step 5 is subjected to contrast learning in a manner as shown in formula (9) by using the ClusterNCE loss;

[0084]

[0085] wherein, represents the stored dictionary visible light pedestrian feature, the stored dictionary infrared pedestrian feature, and τ is a hyperparameter used to control the distance between classes. F vis is the feature of visible light, ir is the feature of infrared, and [F vis , F ir ] is the mixed feature.

[0086] In the embodiment, the value of the hyperparameter τ is 0.05.

[0087] In order to further embody the superiority of the present application, the experimental results are used to illustrate the generalization ability of the visible light and infrared cross-modal pedestrian re-identification method in the present application.

[0088] As shown in formula (10), the feature distance visualization result after using the method of the present application is shown. Figure 2 Before training, the inter-class distance and the intra-class distance distribution of the features are almost coincident; after training, the distance between the inter-class distance distribution and the intra-class distance distribution is significantly increased, and the intra-class distance distribution is on the left side of the coordinate axis, that is, the intra-class distance is small; and the inter-class distance distribution is on the right side of the coordinate axis, indicating that the inter-class distance is large.

[0089] As shown in formula (11), the feature distribution visualization result after using the method of the present application is shown. Figure 3 The distribution diagram of the features extracted by the model before training shows that the distribution of the visible light modal and the infrared modal features is relatively independent, and the features of different identities are mixed together; the distribution diagram of the features extracted by the model after training shows that the feature distance of different identity pedestrians is increased, the distribution of the features of the same identity pedestrians is more compact, and the features of the visible light modal and the infrared modal can be well gathered together.

[0090] As shown in formula (12), the feature distance visualization result after using the method of the present application is shown. Figure 4 and Figure 5As shown, it is the retrieval result from visible light to infrared and from infrared to visible light, the first image on the left side of each row is the query image, and the right side is the result sorted in descending order according to the similarity of the candidate set image and the query image. Among them, the similarity measurement index is the Euclidean distance, the numbers above the image are the sorting numbers, the green color represents the same identity of the image and the query image, and the red color represents different identities.

[0091] The above results show that the generalization capability in the visible light and infrared cross-modal pedestrian re-identification method can be improved by using the present application.

Claims

1. A method for cross-modal pedestrian re-identification based on intermediate domain, the method comprising: The method comprises the following steps, Step 1: generating the features of the intermediate domain by using the source domain and target domain weight in the intermediate domain generator; Step 1.1: in the intermediate domain generator, the weights of the source domain and the target domain are obtained in the manner shown in formula (1); where w s , w t represent the weights of the source domain and the target domain in generating the intermediate domain, respectively, MLP represents a multi-layer perception, and FC represents a fully connected layer; F avg , F gem , and F max represent features subjected to average pooling, generalized average pooling, and maximum pooling, respectively; and δ(·) represents a softmax function. Step 1.2: the weights of the source domain and the target domain in step 1.1 are used in the intermediate domain to form the features of the intermediate domain; F m = w s F s + w t F t (2) wherein w s represents the source domain, w t represents the target domain, F s represents the source domain weight, F t represents the target domain weight; F m is the feature of the intermediate domain; Step 2: the distribution difference between the intermediate domain and the source domain and the target domain is optimized by using the cross-entropy loss, and the distance between the domains is optimized by using the L2 norm; Step 3: the pseudo-label p is assigned to the features of the visible light and the infrared in the target domain optimized in step 2 by using the DBSCAN clustering method, and the features are stored in the manner shown in formula (5) by using the feature storage dictionary; where f (·) denotes a feature extractor, V p , p , VI p denotes the set of features of identity p in the visible light modality, the infrared modality, the visible light and infrared mixed modality, is the set of features of identity p in the visible light modality; is the set of features of identity p in the infrared modality; is the set of features of identity p in the visible light and infrared mixed modality; is the feature storage dictionary in the visible light modality; is the feature storage dictionary in the infrared modality; is the feature storage dictionary in the visible light and infrared mixed modality; Step 4: the graph representation of clustering under the visible light mode and the infrared mode is constructed to calculate the similarity between nodes; Step 5: the clustering feature matching of the weighted bipartite graph of the similarity of the visible light mode and the infrared mode in step 4 is performed; wherein, is an identifier for representing a node and a node belong to the same identity; (i,j) represents the index of the matching relationship to be expanded in the matching matrix Q; represents Q a,b = 1 corresponding eigenvector; d(·) is the Euclidean distance; m represents the matching direction, which is used to ensure that each node is matched; m = vis represents the matching from the visible light modality to the infrared modality, and m = ir represents the matching from the infrared modality to the visible light modality; Step 6: the ClusterNCE loss is used to perform contrastive learning on the clustering feature matching in step 5 in the manner shown in formula (9); wherein, represents a stored dictionary visible light pedestrian feature, a stored dictionary infrared pedestrian feature, and τ is a hyperparameter used to control the separation between classes; F vis is a feature for visible light, F ir is a feature for infrared, [F vis ,F ir ] is a hybrid feature.

2. The method of claim 1, wherein the method is based on an intermediate domain for visible and infrared cross-modal pedestrian re-identification. The implementation method of step 2 is, Step 2.1: the distribution difference between the intermediate domain and the source domain and the target domain is optimized in the manner shown in formula (3) by using the cross-entropy loss; Wherein, k is a domain identifier, indicating that the feature belongs to the source domain or the target domain, i is an index in a small batch, and φ is a classifier. represents a loss function, and a small value indicates a good optimization effect. Step 2.2: the distance between the domains in the feature space is optimized in the manner shown in formula (4) by using the L2 norm; wherein, denotes the minimization of the distances between the source domain and the intermediate domain, the target domain and the intermediate domain.

3. The method of claim 1, wherein the method is based on an intermediate domain for visible and infrared cross-modal pedestrian re-identification. The implementation method of step 4 is, Step 4.1: the graph representation of clustering under the visible light mode and the infrared mode is constructed in the manner shown in formula (6); wherein the visible light modality and the infrared modality are represented by images respectively [I], Y vis , Y ir denotes the number of clusters; Step 4.2: the similarity of the visible light mode and the infrared mode is calculated in the manner shown in formula (7) by using the graph representation in step 4.1; wherein, respectively represent clusters clusters the number of instances in the cluster is represented by the feature mean of the instances of the cluster, represents the feature mean for visible light modality label i; represents the feature mean for infrared modality label i.

4. The method of claim 1, wherein the method is based on an intermediate domain for visible and infrared cross-modal pedestrian re-identification. The value of the hyperparameter τ is 0.05.

Images