A cross-modal pedestrian re-identification method and system based on consistency of authenticity and fake labels

Abstract

The present invention relates to the technical field of image processing and target recognition, and discloses a cross-modal pedestrian re-identification method and system based on the consistency of true and false labels. The method comprises: extracting feature vectors from pedestrian images of two different modes, namely visible light and infrared light, through a deep neural network; calculating the feature vector similarity between the same mode and different modes, constructing the same mode and cross-modal matching matrices, and performing normalization processing to generate the same mode and cross-modal normalized matching matrices; projecting the real label by using the cross-modal normalized matching matrix and the same mode normalized matching matrix to obtain the cross-modal pseudo label; optimizing the Kullback-Leibler (KL) divergence between the real label and the cross-modal pseudo label, thereby optimizing the same mode and cross-modal matching matrices, improving the robustness of the matching matrix to the modality change, and thus improving the accuracy of cross-modal pedestrian re-identification.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and target recognition, and in particular to a cross-modal pedestrian re-identification method and system based on authenticity and false label consistency. Background Art

[0002] Public security intelligent video surveillance works around the clock, taking visible light pedestrian images during the day and infrared light pedestrian images at night. Therefore, visible and infrared cross-modal pedestrian re-identification is of great significance. However, there are significant differences between visible light and infrared light pedestrian images in terms of spectrum, resolution, and contrast, which cause great interference to the modal change robustness of the pedestrian matching matrix. For example, the similarity between visible light pedestrian images and infrared light pedestrian images of the same identity is often much lower than the similarity between visible light pedestrian images of different identities or the similarity between infrared light pedestrian images of different identities, which increases the difficulty of cross-modal pedestrian re-identification. Therefore, in addition to the conventional identity identification optimization, the optimization of the pedestrian matching matrix also needs to consider the modal change robustness. Summary of the invention

[0003] The purpose of the present invention is to solve the problem of ensuring robustness between different modes in the prior art.

[0004] The technical solution adopted by the present invention to solve the technical problem is: to provide a cross-modal pedestrian re-identification method based on authenticity label consistency, comprising the following steps:

[0005] Construct a visible and infrared cross-modal pedestrian re-identification model based on deep neural network;

[0006] The visible and infrared cross-modal pedestrian re-identification model is trained using the true and false label consistency loss function;

[0007] Pedestrian re-identification is achieved using the trained visible and infrared cross-modal pedestrian re-identification model.

[0008] Preferably, the visible and infrared cross-modal pedestrian re-identification model adopts a ResNet50 network, including a network stem Stem, four residual groups, a generalized mean pooling layer, a batch normalization layer and a fully connected layer connected in sequence, wherein the input image outputs a feature vector after passing through a network stem Stem, four residual groups and a generalized mean pooling layer.

[0009] Preferably, the method of training a visible and infrared cross-modal pedestrian re-identification model using a true-false label consistency loss function comprises the following steps:

[0010] Obtain visible light pedestrian images and infrared light pedestrian images as training samples; each image is equipped with an identity label as a true label;

[0011] Input the training sample images into the visible and infrared cross-modal pedestrian re-identification model to obtain the feature vector;

[0012] Calculate the feature vector similarity between the same modality and the feature vector similarity between different modalities, so as to construct the same modality normalized matching matrix and the cross-modality normalized matching matrix respectively;

[0013] The same-modal normalized matching matrix and the cross-modal normalized matching matrix are used to project the true label to obtain the cross-modal pseudo label;

[0014] Optimize the KL divergence between the true label and the cross-modal pseudo-label so that the cross-modal pseudo-label approaches the true label.

[0015] Preferably, the process of constructing the same-modality normalized matching matrix and the cross-modality normalized matching matrix includes the following steps:

[0016] Construct the same-modality matching and cross-modality matching matrices, expressed as:

[0017] ;

[0018] ;

[0019] ;

[0020] ;

[0021] in, represents the same-mode matching matrix from visible light to visible light, express The element corresponding to the i-th row and j-th column of and The cosine similarity between ; represents the same-mode matching matrix from infrared light to infrared light, express The element corresponding to the i-th row and j-th column is equal to and The cosine similarity between ; represents the cross-modal matching matrix from visible light to infrared light, express The element corresponding to the i-th row and j-th column is equal to and The cosine similarity between ; represents the cross-modal matching matrix from infrared light to visible light, express The element corresponding to the i-th row and j-th column is equal to and The cosine similarity between ; and Respectively represent The d-dimensional feature vector corresponding to the jth visible light pedestrian image and the d-dimensional feature vector corresponding to the jth visible light pedestrian image; and Indicates The d-dimensional feature vector corresponding to the infrared pedestrian image and the A d-dimensional feature vector corresponding to an infrared pedestrian image; represents the number of visible light pedestrian images, Indicates the number of infrared pedestrian images; represents the transpose operation, represents the operation of finding the L2 norm of the eigenvector;

[0022] The same-modality matching matrix and the cross-modality matching matrix are normalized and expressed as:

[0023] ;

[0024] ;

[0025] ;

[0026] ;

[0027] in, Represents the same-mode normalized matching matrix from visible light to visible light The element corresponding to the i-th row and j-th column of ; Represents the same-mode normalized matching matrix from infrared light to infrared light The element corresponding to the i-th row and j-th column of ; Represents the cross-modal normalized matching matrix from visible light to infrared light The element corresponding to the i-th row and j-th column of ; Represents the element corresponding to the i-th row and j-th column of the cross-modal normalized matching matrix from infrared light to visible light; exp represents the natural exponential function.

[0028] Preferably, the cross-modal pseudo-label generation process comprises the following steps:

[0029] The cross-modal normalized matching matrix and the same-modal normalized matching matrix are used to project the true label, which is expressed as:

[0030] ;

[0031] ;

[0032] in, and They represent the true labels of visible light images and infrared pedestrian images with the same identity, m represents the number of visible light images, n represents the number of infrared images, and k represents the number of pedestrian identity categories; Indicates infrared true label Cross-modal projection of Represents the real label of visible light Cross-modal projection of

[0033] The cross-modal projection is Softmax normalized and expressed as:

[0034] ;

[0035] ;

[0036] in, Indicates infrared true label Cross-modal pseudo-labels of ; Represents the real label of visible light Cross-modal pseudo-labels of ; express Normalization is performed row by row, that is, the k corresponding elements in each row are normalized.

[0037] Preferably, the KL divergence between the real label and the cross-modal pseudo-label is optimized, the difference between the real label and the cross-modal pseudo-label is measured by using the true-false label consistency loss function L, and the true-false label consistency loss function L is optimized by the gradient descent method, so that the cross-modal pseudo-label approaches the real label;

[0038] The true and false label consistency loss function L is expressed as:

[0039] ;

[0040] in, Represents the true label of visible light pedestrian images The element corresponding to the i-th row and j-th column of ; Indicates infrared true label The element corresponding to the i-th row and j-th column of the cross-modal pseudo label; Indicates the true label of infrared pedestrian images The element corresponding to the i-th row and j-th column of ; Represents the real label of visible light The element corresponding to the i-th row and j-th column of the cross-modal pseudo label.

[0041] Preferably, the method of using the trained visible and infrared cross-modal pedestrian re-identification model to realize pedestrian re-identification is specifically as follows:

[0042] The trained cross-modal pedestrian re-identification model is used to extract the feature vectors of the visible light pedestrian image and infrared light pedestrian image of each pedestrian in the query image set and the registered image set, respectively, to obtain the query image features and the registered image features;

[0043] The Euclidean distance between the query image features and the registered image features is calculated, and the image with the Euclidean distance less than the preset threshold is output as the pedestrian re-identification result.

[0044] The present invention also provides a cross-modal pedestrian re-identification system based on authenticity and fake label consistency, comprising:

[0045] Model building module, which builds a visible and infrared cross-modal pedestrian re-identification model based on deep neural network;

[0046] Model training module, which uses the consistency loss function of true and false labels to train the visible and infrared cross-modal pedestrian re-identification model;

[0047] The pedestrian re-identification module uses the trained visible and infrared cross-modal pedestrian re-identification model to realize pedestrian re-identification.

[0048] The present invention has the following beneficial effects: the present invention innovatively optimizes same-modality and cross-modality matching matrices using real labels and cross-modality pseudo labels in visible light modalities and infrared light modalities under supervised conditions; obtains same-modality and cross-modality normalized matching matrices by calculating the similarity of feature vectors between the same modality and different modalities; then uses the same-modality and cross-modality normalized matching matrices to project the real labels to obtain cross-modality pseudo labels; and then optimizes the Kullback-Leibler (KL) divergence between the real labels and the cross-modality pseudo labels to make the cross-modality pseudo labels closer to the real labels, thereby optimizing the same-modality and cross-modality matching matrices, so that the similarity between visible light pedestrian images and infrared light pedestrian images of the same identity is higher than the similarity between visible light pedestrian images of different identities or the similarity between infrared light pedestrian images of different identities, thereby improving the robustness of the matching matrix to modality changes and improving the accuracy of cross-modality pedestrian re-identification.

[0049] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A method step diagram of an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of model training according to an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of optimizing the consistency of authenticity labels according to an embodiment of the present invention;

[0053] Figure 4 4 is a system structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0054] See also Figure 1 FIG. 1 is a diagram showing steps of a method according to an embodiment of the present invention, comprising the following steps:

[0055] S101, building a visible and infrared cross-modal pedestrian re-identification model based on deep neural network;

[0056] S102, using the authenticity label consistency loss function to train the visible and infrared cross-modal pedestrian re-identification model;

[0057] S103: Use the trained visible and infrared cross-modal pedestrian re-identification model to realize pedestrian re-identification.

[0058] Specifically, the visible and infrared cross-modal pedestrian re-identification model adopts a ResNet50 network, including a network stem Stem, four residual groups (Layer1, Layer2, Layer3, Layer4), a generalized mean pooling (Generalized Mean Pooling, GeP) layer, a batch normalization layer (Batch Normalization, BN) and a fully connected layer (Fully Connected, FC) connected in sequence. The image is input to the ResNet50 network for feature extraction, and the extracted feature vector is output by the generalized mean pooling layer.

[0059] See also Figure 2 FIG. 1 is a schematic diagram of a model training of an embodiment of the present invention, which includes the following steps:

[0060] S201, obtaining visible light pedestrian images and infrared light pedestrian images as training samples; wherein each image is equipped with an identity identifier as a true label;

[0061] S202: Based on the extracted feature vector, a visible and infrared cross-modal pedestrian re-identification model is trained using a true-false label consistency loss function.

[0062] Specifically, the visible and infrared cross-modal pedestrian re-identification model is trained using the true and false label consistency loss function, see Figure 3 As shown, the following steps are included:

[0063] S301, generate a matching matrix. Calculate the similarity of the feature vectors between the same modality and different modalities, and construct the same modality matching matrix and the cross-modality matching matrix, which can be expressed as:

[0064]

[0065] in, represents the same-mode matching matrix from visible light to visible light, express The element corresponding to the i-th row and j-th column of and Cosine similarity between represents the same-mode matching matrix from infrared light to infrared light, express The element corresponding to the i-th row and j-th column is equal to and Cosine similarity between represents the cross-modal matching matrix from visible light to infrared light, express The element corresponding to the i-th row and j-th column is equal to and Cosine similarity between represents the cross-modal matching matrix from infrared light to visible light, express The element corresponding to the i-th row and j-th column is equal to and Cosine similarity between and Respectively represent The d-dimensional feature vector corresponding to the j-th visible light pedestrian image; and Indicates and A d-dimensional feature vector corresponding to an infrared pedestrian image; represents the number of visible light pedestrian images, Indicates the number of infrared pedestrian images; represents the transpose operation, Represents the operation of finding the L2 norm of a eigenvector.

[0066] S302, normalize the matching matrix. Normalize the same-modality and cross-modality feature matching matrices to generate a same-modality normalized matching matrix from visible light to visible light, a same-modality normalized matching matrix from visible light to infrared light, a cross-modality normalized matching matrix from infrared light to visible light, and a cross-modality normalized matching matrix from infrared light to infrared light, as shown in the following formula:

[0067]

[0068] in, Represents the normalized matching matrix from visible light to visible light The element corresponding to the i-th row and j-th column of ; Represents the normalized matching matrix from infrared light to infrared light The element corresponding to the i-th row and j-th column of ; Represents the normalized matching matrix from visible light to infrared light The element corresponding to the i-th row and j-th column of ; Represents the element corresponding to the i-th row and j-th column of the normalized matching matrix from infrared light to visible light; exp represents the natural exponential function.

[0069] S303, generate cross-modal pseudo labels. Use the cross-modal normalized matching matrix and the same-modal normalized matching matrix to project the real label to obtain the cross-modal pseudo label. The processing process of the cross-modal pseudo label generation step is as follows:

[0070]

[0071]

[0072] in, and They represent the true labels corresponding to the visible light and infrared light pedestrian images with the same identity, m represents the number of visible light images, n represents the number of infrared light images, and k represents the number of pedestrian identity categories; Indicates infrared true label The cross-modal projection is first performed by normalizing the cross-modal matching matrix from visible light to infrared light right Perform cross-modal label conversion from infrared light to visible light, and then use the same-modal normalized matching matrix from visible light to visible light obtained by recombinant methods; Represents the real label of visible light The cross-modal projection is first performed by normalizing the cross-modal matching matrix from infrared light to visible light. right Perform cross-modal label conversion from infrared to visible light, and then use the same-modal normalized matching matrix from infrared to infrared Obtained by recombinant.

[0073] The cross-modal projection is Softmax normalized to obtain the cross-modal pseudo label, as shown in the following formula:

[0074]

[0075]

[0076] in, Indicates infrared true label Cross-modal pseudo-labels of ; Represents the real label of visible light Cross-modal pseudo-labels of ; express Normalization is performed row by row, that is, the k corresponding elements in each row are normalized.

[0077] S304, optimize the consistency of true and false labels. Optimize the Kullback-Leibler (KL) divergence between the true label and the cross-modal pseudo label, so that the cross-modal pseudo label approaches the true label. The process of the true and false label consistency optimization step is as follows:

[0078]

[0079] Among them, L represents the consistency loss function of true and false labels; Indicates the true label corresponding to the visible light pedestrian image The element corresponding to the i-th row and j-th column of ; Indicates infrared true label The element corresponding to the i-th row and j-th column of the cross-modal pseudo label; Indicates the true label corresponding to the infrared pedestrian image The element corresponding to the i-th row and j-th column of ; Represents the real label of visible light The element corresponding to the i-th row and j-th column of the cross-modal pseudo label.

[0080] The common gradient descent method is used to optimize L, so that the cross-modal pseudo-label is close to the real label. The same-modal and cross-modal matching matrices are optimized based on the chain rule to improve the robustness of the matching matrix to modal changes, thereby improving the accuracy of cross-modal pedestrian re-identification.

[0081] This embodiment tests a visible and infrared cross-modal pedestrian re-identification method with authenticity label consistency optimization, specifically: using the trained cross-modal pedestrian re-identification model, extract the feature vectors of the visible light pedestrian image and infrared light pedestrian image of each pedestrian in the query image set and the registered image set, respectively, to obtain the query image feature and the registered image feature, based on the extracted feature vector, calculate the Euclidean distance between the query image feature and the feature of each pedestrian image in the registered image, the smaller the Euclidean distance between the two images, the greater the similarity, and then obtain the similarity relationship between the query pedestrian image and each pedestrian image in the registered image. The recognition results are output in descending order according to the similarity. If the image ranked first in the output recognition result has the same identity label as the query image, it means that the query is successful, otherwise it means that the query fails.

[0082] See also Figure 4FIG. 1 is a system structure diagram of an embodiment of the present invention, including:

[0083] A model building module 401 builds a visible and infrared cross-modal pedestrian re-identification model based on a deep neural network;

[0084] A model training module 402 is used to train a visible and infrared cross-modal pedestrian re-identification model using a true and false label consistency loss function;

[0085] The pedestrian re-identification module 403 uses the trained visible and infrared cross-modal pedestrian re-identification model to realize pedestrian re-identification.

[0086] The present invention utilizes homomodal and cross-modal matching matrices to project true labels into cross-modal pseudo-labels, optimizes the Kullback-Leibler (KL) divergence between true labels and cross-modal pseudo-labels, thereby making cross-modal pseudo-labels approach true labels. Based on the chain rule, the homomodal and cross-modal matching matrices are optimized to improve the robustness of the matching matrices to modal changes, thereby improving the accuracy of cross-modal pedestrian re-identification.

[0087] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cross-modal person re-identification method based on authenticity and fake label consistency, characterized in that: The following steps are involved: Construct a visible and infrared cross-modal pedestrian re-identification model based on deep neural network; The visible and infrared cross-modal pedestrian re-identification model is trained using the true and false label consistency loss function; Use the trained visible and infrared cross-modal pedestrian re-identification model to achieve pedestrian re-identification; The method of training a visible and infrared cross-modal pedestrian re-identification model using a true-false label consistency loss function includes the following steps: Obtain visible light pedestrian images and infrared light pedestrian images as training samples; each image is equipped with an identity label as a true label; Input the training sample images into the visible and infrared cross-modal pedestrian re-identification model to obtain the feature vector; Calculate the feature vector similarity between the same modality and the feature vector similarity between different modalities, so as to construct the same modality normalized matching matrix and the cross-modality normalized matching matrix respectively; The same-modal normalized matching matrix and the cross-modal normalized matching matrix are used to project the true label to obtain the cross-modal pseudo label; Optimize the KL divergence between the true label and the cross-modal pseudo-label, so that the cross-modal pseudo-label approaches the true label; The process of constructing the same-modality normalized matching matrix and the cross-modality normalized matching matrix includes the following steps: Construct the same-modality matching and cross-modality matching matrices, expressed as: in, represents the same-mode matching matrix from visible light to visible light, M VV (i,j) represents M VV The element corresponding to the i-th row and j-th column of is equal to v i and v j The cosine similarity between ; represents the same-mode matching matrix from infrared light to infrared light, M TT (i,j) represents M TT The element corresponding to the i-th row and j-th column is equal to t i and t j The cosine similarity between ; represents the cross-modal matching matrix from visible light to infrared light, M VT (i,j) represents M VT The element corresponding to the i-th row and j-th column is equal to v i and t j The cosine similarity between ; represents the cross-modal matching matrix from infrared light to visible light, M TV (i,j) represents M TV The element corresponding to the i-th row and j-th column is equal to t i and v j The cosine similarity between i and v j They represent the d-dimensional feature vector corresponding to the i-th visible light pedestrian image and the d-dimensional feature vector corresponding to the j-th visible light pedestrian image respectively; t i and t j represents the d-dimensional feature vector corresponding to the i-th infrared pedestrian image and the d-dimensional feature vector corresponding to the j-th infrared pedestrian image; m represents the number of visible light pedestrian images, n represents the number of infrared pedestrian images; T represents the transposition operation, |||| represents the operation of finding the L2 norm of the feature vector; The same-modality matching matrix and the cross-modality matching matrix are normalized and expressed as: Among them, P VV (i, j) represents the same-mode normalized matching matrix P from visible light to visible light VV The element corresponding to the i-th row and j-th column of TT (i, j) represents the normalized matching matrix P of infrared light to infrared light in the same mode TT The element corresponding to the i-th row and j-th column of VT (i, j) represents the cross-modal normalized matching matrix P from visible light to infrared light VT The element corresponding to the i-th row and j-th column of TV (i,j) represents the element corresponding to the i-th row and j-th column of the cross-modal normalized matching matrix from infrared light to visible light; exp represents the natural exponential function.

2. The cross-modal person re-identification method based on authenticity and false label consistency according to claim 1 is characterized in that: The visible and infrared cross-modal pedestrian re-identification model adopts a ResNet50 network, including a network stem Stem, four residual groups, a generalized mean pooling layer, a batch normalization layer and a fully connected layer connected in sequence, wherein the input image passes through a network stem Stem, four residual groups and a generalized mean pooling layer to output a feature vector.

3. The cross-modal pedestrian re-identification method based on authenticity and fake label consistency according to claim 1 is characterized in that: The cross-modal pseudo-label generation process includes the following steps: The cross-modal normalized matching matrix and the same-modal normalized matching matrix are used to project the true label, which is expressed as: in, and They represent the true labels of visible light pedestrian images and infrared pedestrian images with the same identity, m represents the number of visible light pedestrian images, n represents the number of infrared pedestrian images, and k represents the number of pedestrian identity categories; The true label Y represents the infrared pedestrian image T Cross-modal projection of Represents the true label Y of the visible light pedestrian image V Cross-modal projection of The cross-modal projection is Softmax normalized and expressed as: in, The true label Y represents the infrared pedestrian image T Cross-modal pseudo-labels of ; Represents the true label Y of the visible light pedestrian image V dim=1 means that Softmax normalization is performed row by row, that is, the k corresponding elements of each row are normalized.

4. The cross-modal pedestrian re-identification method based on authenticity and false label consistency according to claim 1 is characterized in that: The KL divergence between the real label and the cross-modal pseudo-label is optimized, the difference between the real label and the cross-modal pseudo-label is measured by using the true-false label consistency loss function L, and the true-false label consistency loss function L is optimized by the gradient descent method, so that the cross-modal pseudo-label approaches the real label; The true and false label consistency loss function L is expressed as: Among them, Y V (i,j) represents the true label Y of the visible light pedestrian image V The element corresponding to the i-th row and j-th column of ; The true label Y represents the infrared pedestrian image T The element corresponding to the i-th row and j-th column of the cross-modal pseudo label of Y T (i,j) represents the true label Y of the infrared pedestrian image T The element corresponding to the i-th row and j-th column of ; Represents the true label Y of the visible light pedestrian image V The element corresponding to the i-th row and j-th column of the cross-modal pseudo label.

5. The cross-modal pedestrian re-identification method based on authenticity and fake label consistency according to claim 1 is characterized in that: The method of using the trained visible and infrared cross-modal pedestrian re-identification model to realize pedestrian re-identification is specifically as follows: The trained cross-modal pedestrian re-identification model is used to extract the feature vectors of the visible light pedestrian image and infrared light pedestrian image of each pedestrian in the query image set and the registered image set, respectively, to obtain the query image features and the registered image features; The Euclidean distance between the query image features and the registered image features is calculated, and the image with the Euclidean distance less than the preset threshold is output as the pedestrian re-identification result.

6. A cross-modal person re-identification system based on authenticity and fake label consistency, characterized in that: include: Model building module, which builds a visible and infrared cross-modal pedestrian re-identification model based on deep neural network; Model training module, which uses the consistency loss function of true and false labels to train the visible and infrared cross-modal pedestrian re-identification model; The pedestrian re-identification module uses the trained visible and infrared cross-modal pedestrian re-identification model to realize pedestrian re-identification; the training of the visible and infrared cross-modal pedestrian re-identification model using the true and false label consistency loss function includes the following steps: Obtain visible light pedestrian images and infrared light pedestrian images as training samples; each image is equipped with an identity label as a true label; Input the training sample images into the visible and infrared cross-modal pedestrian re-identification model to obtain the feature vector; Calculate the feature vector similarity between the same modality and the feature vector similarity between different modalities, so as to construct the same modality normalized matching matrix and the cross-modality normalized matching matrix respectively; The same-modal normalized matching matrix and the cross-modal normalized matching matrix are used to project the true label to obtain the cross-modal pseudo label; Optimize the KL divergence between the true label and the cross-modal pseudo-label, so that the cross-modal pseudo-label approaches the true label; The process of constructing the same-modality normalized matching matrix and the cross-modality normalized matching matrix includes the following steps: Construct the same-modality matching and cross-modality matching matrices, expressed as: in, represents the same-mode matching matrix from visible light to visible light, M VV (i,j) represents M VV The element corresponding to the i-th row and j-th column of is equal to v i and v j The cosine similarity between ; represents the same-mode matching matrix from infrared light to infrared light, M TT (i,j) represents M TT The element corresponding to the i-th row and j-th column is equal to t i and t j The cosine similarity between ; represents the cross-modal matching matrix from visible light to infrared light, M VT (i,j) represents M VT The element corresponding to the i-th row and j-th column is equal to v i and t j The cosine similarity between ; represents the cross-modal matching matrix from infrared light to visible light, M TV (i,j) represents M TV The element corresponding to the i-th row and j-th column is equal to t i and v j The cosine similarity between i and v j They represent the d-dimensional feature vector corresponding to the i-th visible light pedestrian image and the d-dimensional feature vector corresponding to the j-th visible light pedestrian image respectively; t i and t j represents the d-dimensional feature vector corresponding to the i-th infrared pedestrian image and the d-dimensional feature vector corresponding to the j-th infrared pedestrian image; m represents the number of visible light pedestrian images, n represents the number of infrared pedestrian images; T represents the transposition operation, |||| represents the operation of finding the L2 norm of the feature vector; The same-modality matching matrix and the cross-modality matching matrix are normalized and expressed as: Among them, P VV (i, j) represents the same-mode normalized matching matrix P from visible light to visible light VV The element corresponding to the i-th row and j-th column of TT (i, j) represents the normalized matching matrix P of infrared light to infrared light in the same mode TT The element corresponding to the i-th row and j-th column of VT (i, j) represents the cross-modal normalized matching matrix P from visible light to infrared light VT The element corresponding to the i-th row and j-th column of TV (i,j) represents the element corresponding to the i-th row and j-th column of the cross-modal normalized matching matrix from infrared light to visible light; exp represents the natural exponential function.

Images