Visible light and infrared cross-modal pedestrian re-identification method based on intermediate domain

Abstract

The invention discloses a visible light and infrared cross-modal pedestrian re-identification method based on an intermediate domain, and belongs to the technical field of computer vision. The method comprises the following steps: 1, generating features of an intermediate domain by utilizing weights of a source domain and a target domain in an intermediate domain generator; 2, optimizing the distribution difference among the intermediate domain, the source domain and the target domain by using cross entropy loss, and optimizing the distance between the domains by using an L2 norm; 3, using a DBSCAN clustering method to allocate pseudo labels p to the optimized visible light and infrared features in the target domain, and using a feature storage dictionary to store the features; 4, graph representation of clustering in the visible light mode and the infrared mode is constructed and used for calculating the similarity of the nodes; 5, performing weighted bipartite graph clustering feature matching on the similarity of the visible light modality and the infrared modality; and 6, using ClusterNCE loss to carry out comparative learning on clustering feature matching. Compared with the prior art, the generalization ability in the visible light and infrared cross-modal pedestrian re-identification method can be improved.

Description

Technical Field

[0001] The invention relates to a visible light and infrared cross-modal pedestrian re-identification method based on an intermediate domain, and belongs to the technical field of computer vision. Background Art

[0002] The task of cross-modal pedestrian re-identification between visible light and infrared is an important challenge in the field of computer vision and is of great significance to applications such as all-weather video surveillance, intelligent security and crowd flow analysis.

[0003] Although some progress has been made in the visible light and infrared cross-modal pedestrian re-identification methods based on supervised learning, the application scope of these methods is limited due to the high cost of obtaining high-quality annotated data. In contrast, unsupervised learning without labels is more practical in visible light and infrared cross-modal pedestrian re-identification.

[0004] However, most current studies focus on the performance of the model on a specific dataset and ignore the study of the model's generalization ability, resulting in a significant decrease in algorithm performance when the data domain changes, thus limiting its practicality.

[0005] Therefore, when the data domain changes, how to improve the generalization ability of visible light and infrared cross-modal pedestrian re-identification methods has become an urgent problem to be solved. Summary of the invention

[0006] The purpose of the present invention is to solve the technical problem of low generalization ability in the existing visible light and infrared cross-modal pedestrian re-identification method. A visible light and infrared cross-modal pedestrian re-identification method based on the intermediate domain is proposed.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] A visible light and infrared cross-modal pedestrian re-identification method based on an intermediate domain of the present invention comprises the following steps:

[0009] Step 1: Generate the features of the intermediate domain using the weights of the source domain and the target domain in the intermediate domain generator;

[0010] Step 1.1: In the intermediate domain generator, the weights of the source domain and the target domain are obtained as shown in formula (1);

[0011]

[0012] Among them, w s 、w t They represent the weights of the source domain and the target domain in generating the intermediate domain, MLP represents a multi-layer perceptron, and FC represents a fully connected layer. avg 、F gem 、Fmax They represent the features after average pooling, generalized average pooling, and maximum pooling, respectively. δ(·) represents the softmax function.

[0013] Step 1.2: The weights of the source domain and the target domain described in step 1.1 are applied to the intermediate domain to form the features of the intermediate domain.

[0014] F m =w s F s +w t F t (2)

[0015] Among them, w s represents the source domain, w t represents the target domain, F s represents the source domain weight, F t represents the target domain weight. m is the characteristic of the middle domain;

[0016] Step 2: Use cross entropy loss to optimize the distribution difference between the intermediate domain and the source domain and target domain, and use L2 norm to optimize the distance between domains;

[0017] Step 2.1: Use the cross entropy loss to optimize the distribution difference between the intermediate domain and the source domain and the target domain as shown in formula (3);

[0018]

[0019] where k is the domain identifier, indicating that the feature belongs to the source domain or the target domain, i is the index in a mini-batch, and φ is the classifier. Represents the loss function, and a smaller value means a better optimization effect.

[0020] Step 2.2: Use the L2 norm to optimize the distance between domains in the feature space as shown in formula (4);

[0021]

[0022] in, It means minimizing the distance between the source domain and the intermediate domain, and between the target domain and the intermediate domain.

[0023] Step 3: Use the DBSCAN clustering method to assign pseudo labels p to the visible light and infrared features in the target domain optimized in step 2, and use the feature storage dictionary to store the features as shown in formula (5).

[0024]

[0025] Where f(·) represents the feature extractor, Vp ,I p VI p represents the feature set with identity p in the visible light modality, infrared modality, and visible light and infrared mixed modality, is the feature set with identity p in the visible light mode; The feature set with identity p in infrared modality; is the feature set with identity p in the mixed mode of visible light and infrared; Store dictionary for features in visible light mode; Store dictionary for features in infrared mode; A dictionary for storing features in the mixed mode of visible light and infrared;

[0026] Step 4: Construct a graph representation of clustering in the visible light mode and infrared mode to calculate the similarity between nodes;

[0027] Step 4.1: Construct a graph representation of clustering in the visible light mode and infrared mode as shown in formula (6);

[0028]

[0029] The graphical representations of the visible light mode and infrared mode are [I],Y vis , Y ir Indicates the number of clusters.

[0030] Step 4.2: Calculate the similarity between the visible light mode and the infrared mode using the method shown in formula (7) for the graph representation described in step 4.1;

[0031]

[0032] in, Respectively represent clusters Clustering The number of instances in the cluster is represented by the feature mean of the instance. represents the feature mean of the visible light modality label i; represents the feature mean of infrared modality label i;

[0033] Step 5: Perform clustering feature matching of weighted bipartite graph on the similarity between the visible light modality and the infrared modality described in step 4.

[0034]

[0035] in, Identifier used to represent the node and nodes Whether they belong to the same identity. (i, j) represents the index of the matching relationship to be expanded in the matching matrix Q; Indicates Q a，b =1; d(·) is the Euclidean distance; m represents the matching direction, which is used to ensure that each node is matched. m=vis represents the matching from the visible light mode to the infrared mode, and m=ir represents the matching from the infrared mode to the visible light mode;

[0036] Step 6: Use ClusterNCE loss to perform comparative learning on the cluster feature matching described in step 5 in the manner shown in formula (9);

[0037]

[0038]

[0039] in, Represents the storage dictionary of visible light pedestrian features, The dictionary stores infrared pedestrian features, and τ is a hyperparameter used to control the spacing between categories. vis is the characteristic of visible light, F ir is the characteristic of infrared, [F vis , F ir ] is a mixed feature;

[0040] Beneficial effects:

[0041] 1. To address the problem of differences between domains, the hidden representations of the source domain and the target domain features are mixed to generate an intermediate domain, so that the model learns a more consistent feature representation, thereby reducing the differences between domains. To address the problem of modal differences within a domain, supervised learning is used to optimize the labeled source domain. For the unlabeled target domain, clustering of mixed modalities is added on the basis of clustering different modalities separately to indirectly establish cross-modal correspondence. Weighted bipartite graph matching is used to further align the visible light modality and the infrared modality to reduce modal differences.

[0042] 2. Compared with the prior art, the method of the present invention can still maintain a high recognition accuracy when the data domain changes, and is practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the principle of the present invention;

[0044] Figure 2 This is the feature distance visualization result after using the method of the present invention;

[0045] Figure 3 The feature distribution visualization result after using the method of the present invention;

[0046] Figure 4 The visible light and infrared search results of the method of the present invention;

[0047] Figure 5 These are the infrared and visible light search results of the method of the present invention. DETAILED DESCRIPTION

[0048] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the protection scope of the present invention.

[0049] Example

[0050] like Figure 1 As shown, the visible light and infrared cross-modal pedestrian re-identification method based on the intermediate domain of the present invention is specifically implemented in the following steps:

[0051] First, the source domain and the target domain are sent to the network respectively. In this embodiment, the source domain is the SYSU-MM01 dataset and the target domain is the LLCM dataset. The input image is a visible light image and an infrared image with a size of 288×144 pixels.

[0052] Step 1: Generate the features of the intermediate domain using the weights of the source domain and the target domain in the intermediate domain generator;

[0053] Step 1.1: In the intermediate domain generator, the weights of the source domain and the target domain are obtained as shown in formula (1);

[0054]

[0055] Among them, w s 、w t They represent the weights of the source domain and the target domain in generating the intermediate domain, MLP represents a multi-layer perceptron, and FC represents a fully connected layer. avg 、F gem 、F max They represent the features after average pooling, generalized average pooling, and maximum pooling, respectively. δ(·) represents the softmax function.

[0056] Step 1.2: The weights of the source domain and the target domain described in step 1.1 are applied to the intermediate domain to form the features of the intermediate domain.

[0057] F m =w s F a +w t F t (2)

[0058] Among them, w srepresents the source domain, w t represents the target domain, F s represents the source domain weight, F t represents the target domain weight. m is the characteristic of the middle domain;

[0059] Step 2: Use cross entropy loss to optimize the distribution difference between the intermediate domain and the source domain and target domain, and use L2 norm to optimize the distance between domains;

[0060] Step 2.1: Use the cross entropy loss to optimize the distribution difference between the intermediate domain and the source domain and the target domain as shown in formula (3);

[0061]

[0062] where k is the domain identifier, indicating that the feature belongs to the source domain or the target domain, i is the index in a mini-batch, and φ is the classifier. Represents the loss function, and a smaller value means a better optimization effect.

[0063] Step 2.2: Use the L2 norm to optimize the distance between domains in the feature space as shown in formula (4);

[0064]

[0065] in, It means minimizing the distance between the source domain and the intermediate domain, and between the target domain and the intermediate domain.

[0066] Step 3: Use the DBSCAN clustering method to assign pseudo labels p to the visible light and infrared features in the target domain optimized in step 2, and use the feature storage dictionary to store the features as shown in formula (5).

[0067]

[0068]

[0069] Where f(·) represents the feature extractor, V p ,I p VI p represents the feature set with identity p in the visible light modality, infrared modality, and visible light and infrared mixed modality, is the feature set with identity p in the visible light mode; The feature set with identity p in infrared modality; is the feature set with identity p in the mixed mode of visible light and infrared; Store dictionary for features in visible light mode; Store dictionary for features in infrared mode; A dictionary for storing features in the mixed mode of visible light and infrared;

[0070] In the embodiment, the maximum clustering distance of DBSCAN is set to 0.6, and the minimum number of instances contained in each cluster is set to 4.

[0071] Step 4: Construct a graph representation of clustering in the visible light mode and infrared mode to calculate the similarity between nodes;

[0072] Step 4.1: Construct a graph representation of clustering in the visible light mode and infrared mode as shown in formula (6);

[0073]

[0074] The graphical representations of the visible light mode and infrared mode are [I],Y vis , Y ir Indicates the number of clusters.

[0075] Step 4.2: Calculate the similarity between the visible light mode and the infrared mode using the method shown in formula (7) for the graph representation described in step 4.1;

[0076]

[0077] in, Respectively represent clusters Clustering The number of instances in the cluster is represented by the feature mean of the instance. represents the feature mean of the visible light modality label i; represents the feature mean of infrared modality label i;

[0078] Step 5: Perform clustering feature matching of weighted bipartite graph on the similarity between the visible light modality and the infrared modality described in step 4.

[0079]

[0080]

[0081] in, Identifier used to represent the node and nodes Whether they belong to the same identity. (i, j) represents the index of the matching relationship to be expanded in the matching matrix Q; Indicates Q a，b=1; d(·) is the Euclidean distance; m represents the matching direction, which is used to ensure that each node is matched. m=vis represents the matching from the visible light mode to the infrared mode, and m=ir represents the matching from the infrared mode to the visible light mode;

[0082] In the embodiment, Q m =1 indicates the same identity, Q m = 0 indicates different identities. m indicates the matching direction, which is used to ensure that each node is matched.

[0083] Step 6: Use ClusterNCE loss to perform comparative learning on the cluster feature matching described in step 5 in the manner shown in formula (9);

[0084]

[0085] in, Represents the storage dictionary of visible light pedestrian features, The dictionary stores infrared pedestrian features, and τ is a hyperparameter used to control the spacing between categories. vis is the characteristic of visible light, F ir is the characteristic of infrared, [F vis , F ir ] is a mixed feature;

[0086] In the embodiment, the value of the hyperparameter τ is 0.05.

[0087] In order to further demonstrate the superiority of the present invention, experimental results are used to illustrate the generalization ability of the visible light and infrared cross-modal pedestrian re-identification method in the present invention.

[0088] like Figure 2 As shown, it is the visualization result of feature distance after using the method of the present invention; before training, the distribution of inter-class distance and intra-class distance of features almost overlap; after training, the distance between the distribution of inter-class distance and intra-class distance increases significantly, and the intra-class distance is distributed on the left side of the coordinate axis, that is, the intra-class distance is small; while the inter-class distance is distributed on the right side of the coordinate axis, indicating that the inter-class distance is large.

[0089] like Figure 3 As shown in the figure, it is the visualization result of feature distribution after using the method of the present invention. In the distribution diagram of features extracted by the model before training, the distribution of visible light modality and infrared modality features are relatively independent, and features of different identities are mixed together. In the distribution diagram of features extracted by the model after training, the feature distance of pedestrians with different identities is increased, the distribution of features of pedestrians with the same identity is more compact, and the features of visible light modality and infrared modality can be well clustered together.

[0090] like Figure 4 and Figure 5As shown in the figure, the search results from visible light to infrared and infrared to visible light are shown. The first image on the left side of each row is the query image, and the right side is the result sorted from large to small according to the similarity between the candidate set image and the query image. The similarity measurement indicator is the Euclidean distance, and the number above the image is the sorting number. Green means that the image and the query image are pedestrians of the same identity, and red means different identities.

[0091] The above results show that the use of the present invention can improve the generalization ability of the visible light and infrared cross-modal pedestrian re-identification method.

Claims

1. A visible light and infrared cross-modal pedestrian re-identification method based on intermediate domain, characterized by: The following steps are included: Step 1: Generate the features of the intermediate domain using the weights of the source domain and the target domain in the intermediate domain generator; Step 2: Use cross entropy loss to optimize the distribution difference between the intermediate domain and the source domain and target domain, and use L2 norm to optimize the distance between domains; Step 3: Use the DBSCAN clustering method to assign pseudo labels p to the visible light and infrared features in the target domain optimized in step 2, and use the feature storage dictionary to store the features as shown in formula (5); Where f(·) represents the feature extractor, V p ,I p VI p represents the feature set with identity p in the visible light modality, infrared modality, and visible light and infrared mixed modality, is the feature set with identity p in the visible light mode; The feature set with identity p in infrared modality; is the feature set with identity p in the mixed mode of visible light and infrared; Store dictionary for features in visible light mode; Store dictionary for features in infrared mode; A dictionary for storing features in the mixed mode of visible light and infrared; Step 4: Construct a graph representation of clustering in the visible light mode and infrared mode to calculate the similarity between nodes; Step 5: Perform cluster feature matching of weighted bipartite graph on the similarity between the visible light modality and the infrared modality described in step 4; in, Identifier used to represent the node and nodes Whether they belong to the same identity; (i, j) represents the index of the matching relationship to be expanded in the matching matrix Q; Indicates Q a,b =1; d(·) is the Euclidean distance; m represents the matching direction, which is used to ensure that each node is matched; m=vis represents the matching from the visible light mode to the infrared mode, and m=ir represents the matching from the infrared mode to the visible light mode; Step 6: Use ClusterNCE loss to perform comparative learning on the cluster feature matching described in step 5 in the manner shown in formula (9); in, Represents the storage dictionary of visible light pedestrian features, Dictionary infrared pedestrian features are stored, τ is a hyperparameter used to control the spacing between categories; F vis is the characteristic of visible light, F ir is the characteristic of infrared, [F vis ,F ir ] is a mixed feature.

2. The visible light and infrared cross-modal pedestrian re-identification method based on the intermediate domain as claimed in claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: In the intermediate domain generator, the weights of the source domain and the target domain are obtained as shown in formula (1); Among them, w s 、w t They represent the weights of the source domain and the target domain in generating the intermediate domain, MLP represents a multi-layer perceptron, and FC represents a fully connected layer; F avg 、F gem 、F max They represent the features after average pooling, generalized average pooling, and maximum pooling respectively; δ(·) represents the softmax function; Step 1.2: The weights of the source domain and the target domain described in step 1.1 are applied to the intermediate domain to form the features of the intermediate domain. F m =w s F s +w t F t (2) Among them, w s represents the source domain, w t represents the target domain, F s represents the source domain weight, F t represents the target domain weight; F m Characteristics of the middle domain.

3. The visible light and infrared cross-modal pedestrian re-identification method based on the intermediate domain as claimed in claim 1, characterized in that: Step 2 is implemented as follows: Step 2.1: Use the cross entropy loss to optimize the distribution difference between the intermediate domain and the source domain and the target domain as shown in formula (3); Where k is the domain identifier, indicating that the feature belongs to the source domain or the target domain, i is the index in a mini-batch, and φ is the classifier; Represents the loss function, and a small value indicates a good optimization effect; Step 2.2: Use the L2 norm to optimize the distance between domains in the feature space as shown in formula (4); in, It means minimizing the distance between the source domain and the intermediate domain, and between the target domain and the intermediate domain.

4. The visible light and infrared cross-modal pedestrian re-identification method based on the intermediate domain as claimed in claim 1, characterized in that: Step 4 is implemented as follows: Step 4.1: Construct a graph representation of clustering in the visible light mode and infrared mode as shown in formula (6); The graphical representations of the visible light mode and infrared mode are [I],Y vis , Y ir Indicates the number of clusters; Step 4.2: Calculate the similarity between the visible light mode and the infrared mode using the method shown in formula (7) for the graph representation described in step 4.1; in, Respectively represent clusters Clustering The number of instances in the cluster is represented by the feature mean of the instance. represents the feature mean of the visible light modality label i; Represents the feature mean of infrared modality label i.

5. The visible light and infrared cross-modal pedestrian re-identification method based on the intermediate domain as claimed in claim 1, characterized in that: The value of the hyperparameter τ is 0.05.

Images