Unsupervised visible-infrared person re-identification method based on cyclic pair-wise identity learning

Abstract

The present application belongs to the field of computer vision, artificial intelligence and intelligent security technology, and specifically relates to an unsupervised visible light-infrared personnel re-identification method based on cyclic pair identity learning. In the training stage, a backbone network is used to extract multi-modal features from pedestrian images, and a cross-modal similarity matrix is calculated according to the multi-modal features. The cross-modal similarity matrix is projected into a pair matrix through a pair matrix projection network. A multi-modal relationship matrix is calculated by fusing the pair matrix and the cross-modal identity matching information. An invertible mapping model is used to align the modes, and the network and model are trained using loss in the training stage. In the identification stage, the cosine similarity of the query image feature set and all features in the gallery feature set is calculated, and the images corresponding to the top-ranked features are taken as the cross-modal identity re-identification results of the query image. The present application can realize accurate matching and identity re-identification of visible light and infrared modal pedestrian images under unsupervised conditions.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, artificial intelligence and intelligent security technology, specifically an unsupervised visible light-infrared personnel re-identification method based on cyclic pairwise identity learning. Background Technology

[0002] People re-identification technology aims to identify pedestrians with the same identity across different cameras, at different times, or in different scenarios. With the increasing demand for all-weather security, visible light-infrared people re-identification (VI-ReID) technology has become crucial because it can utilize surveillance information from both daytime (visible light) and nighttime (infrared).

[0003] Existing visible light-infrared personnel re-identification technologies mainly fall into two categories:

[0004] 1. Supervised cross-modal methods: These methods rely on training with a large amount of paired labeled data and reduce modal differences by designing cross-modal feature alignment networks (such as generative adversarial networks or feature projection networks). However, these methods are costly to annotate and are difficult to apply in real-world scenarios.

[0005] 2. Unsupervised cross-modal methods: These methods typically first use clustering algorithms to generate pseudo-labels for unlabeled data, and then perform feature learning and alignment based on these pseudo-labels. However, existing methods have significant drawbacks: ① The clustering process is easily affected by modal differences, resulting in noisy cross-modal pseudo-labels; ② When aligning modal features, strategies such as direct fusion or forced mapping are often used, which can easily destroy or obscure the inherent feature information within each modality that is crucial for identity discrimination, causing the model to be unable to effectively distinguish pedestrians of different identities.

[0006] Therefore, visible light-infrared person re-identification technology faces two major challenges: first, there are significant modal differences between visible light and infrared images; second, it is difficult to obtain a large amount of accurately labeled cross-modal pedestrian data in practical applications, i.e., there is a problem of missing labels. Therefore, there is an urgent need for a visible light-infrared person re-identification method that can effectively align cross-modal features under unsupervised conditions, while finely preserving and utilizing identity discrimination information. Summary of the Invention

[0007] To address the unavoidable modal differences and lack of labeled data in visible light-infrared person re-identification, this invention provides an unsupervised visible light-infrared person re-identification method based on cyclic pairwise identity learning, which achieves accurate matching and identity re-identification of visible light and infrared modal pedestrian images under unsupervised conditions.

[0008] This invention is implemented using the following technical solution: an unsupervised visible light-infrared person re-identification method based on cyclic pairwise identity learning, including a training phase and an identification phase. In the training phase, firstly, a backbone network is used... Multimodal features are extracted from pedestrian images, and then a cross-modal similarity matrix is ​​calculated based on these features. Through pairwise matrix projection networks Cross-modal similarity matrix Projected onto pairwise matrices Then, by merging them into pairs of matrices Using cross-modal identity matching information to calculate the multimodal relationship matrix Finally, the invertible mapping model is used. Align modalities and train the backbone network using loss during the training phase. Paired matrix projection networks and invertible mapping model In the recognition stage, the trained network and model are used to construct a library feature set for the target image library and a query image feature set for the input query image. The cosine similarity between the query image feature set and all features in the library feature set is calculated. The features are sorted from high to low according to the cosine similarity value, and the images corresponding to the top L features are taken as the cross-modal identity re-recognition results of the query image.

[0009] The aforementioned unsupervised visible light-infrared personnel re-identification method based on cyclic pairwise identity learning includes the following steps in the training phase:

[0010] Step S100: Dual-modal feature extraction: Input visible light image set and infrared image set ,in, Indicates the first Visible light image, Indicates the number of visible light images. Indicates the first 1 infrared image, Indicates the number of infrared images;

[0011] Use backbone network Multimodal features were extracted separately to obtain the visible light feature set. and infrared feature set ;in, Indicates the first Features of a visible light image Indicates the first Features of an infrared image;

[0012] Visible light feature set and infrared feature set Clustering algorithms are applied separately to generate a set of cluster centers for each modality, denoted as follows: and ,in, Represents a visible light cluster set. Represents the first visible light Cluster centers, Represents an infrared cluster set. Indicating the infrared first Cluster centers;

[0013] Step S200: Calculate the identity loss by jointly analyzing cross-modal cluster centers and pedestrian features. The loss function is formalized as follows: , ,in, This indicates loss of visible light identity. Indicates loss of infrared identity. Indicates and Visible light cluster centers with the same pseudo-label Indicates and Infrared cluster centers with the same pseudo-labels Indicates transpose calculation. This indicates the number of cluster centers for visible light. This indicates the number of cluster centers in the infrared spectrum. It is a temperature parameter; ultimately, it will affect the multimodal identity loss. The definition is as follows: ;

[0014] S210: Iterative Identity Relationship Matching: First, define the matching matrix. Matching elements in the matrix Represents the first visible light Cluster centers With infrared Cluster centers Determine if a match is found, and then solve the following optimization problem: , , , ,in, Let be the cost matrix, and its elements , Cosine similarity; iteratively refine the matching matrix. The final output is a visible-to-infrared cross-modal matching dictionary. Infrared-to-visible cross-modal matching dictionary Used to derive visible light false tags and infrared fake tags ;

[0015] S220: Pairwise Matrix Learning and Optimization

[0016] 1) Calculate the cross-modal similarity matrix : ,in, Represents the cross-modal similarity matrix The elements in The L2 norm of a vector;

[0017] 2) Cross-modal similarity matrix is ​​mapped through network mapping Projected into a more identity-distinguishing pairwise matrix middle, ,in, Represents a pairwise matrix, where each element in the pairwise matrix is ​​a pairwise matrix. express and Probability estimate of belonging to the same identity; It is a pairwise matrix projection network;

[0018] 3) Training a pairwise matrix projection network The pairwise matrix loss is defined as: , , ,in, Represents pairwise matrix loss. Indicates the loss due to false labels. and , Indicates relative order loss;

[0019] S230: Constructing a reliable multimodal relation matrix To suppress noise, pairwise matrices are used. Threshold filtering is performed to obtain a cleaner multimodal relation matrix. : ,in, For the preset threshold, Representing the multimodal relation matrix The elements in Indicates visible light pseudo-label The first in One element, Indicating infrared fake tags The first in Elements; based on a multimodal relation matrix Calculate the visible light confidence vector of each feature and its cross-modal identity relationship. and infrared confidence vector : ,in, Represents the visible light confidence vector The first in One element, Infrared confidence vector The first in One element;

[0020] S240: Calculate cross-modal identity alignment loss : , , ,in, Visible light to infrared cross-modal matching dictionary Found, and The corresponding cross-modal cluster centers, Infrared-to-visible cross-modal matching dictionary Found, and The corresponding cross-modal cluster centers, This indicates the visible light to infrared alignment loss. This indicates the alignment loss between infrared and visible light;

[0021] Step S300: Cross-modal cyclic distribution alignment based on an invertible mapping model, introducing an invertible mapping model. The invertible mapping model and its inverse function The definition is as follows: , ,in, Represents the visible light to infrared feature set. Represents the infrared-to-visible light feature set;

[0022] Reversible mode alignment loss The design is as follows: by using feature alignment loss and self-supervised consistency constraints, identity information is ensured to remain intact during the transformation process. ,

[0023] Feature alignment loss : ,in, Visible light variable infrared feature set The first in One element, Infrared to Visible Light Feature Set The first in Each element; the feature alignment loss drives cross-modal feature pairs with high matching probabilities to be close to each other in the feature space;

[0024] Self-supervised consistency constraint : This self-supervised consistency constraint requires features to maintain consistency after going through a cycle of "modal transformation-inverse transformation" to ensure that identity discrimination information is not lost during the reversible transformation process;

[0025] Step S400: Joint training and optimization of the model, final total loss It is the weighted sum of the above losses: ,in, and To balance the hyperparameters;

[0026] An alternation strategy is used during the training phase:

[0027] 1) Pairwise matrix projection network Using pairwise matrix loss Independent training;

[0028] 2) Backbone Network and invertible mapping model Total loss Joint training.

[0029] The aforementioned unsupervised visible light-infrared personnel re-identification method based on cyclic pairwise identity learning specifically includes the following steps in the identification phase:

[0030] Step S500: Identity Re-identification:

[0031] S510: Image Library Feature Processing:

[0032] S511: For the target image database, pre-process and store features in batches, then input the images from the target image database into the trained backbone network. It also outputs the feature set of the image library: if it is a visible light image library, then output... If it is an infrared image library, then output... ;

[0033] S512: Constructing a cross-modal similarity matrix :

[0034] If it is a visible light image library: select the infrared feature set from the training phase. Using the visible light image library feature set as a reference feature set, the similarity between the reference feature set and the visible light image library feature set is calculated, and a cross-modal similarity matrix is ​​constructed. ;

[0035] If it is an infrared image library: select the visible light feature set from the training phase. As a reference feature set, the similarity between the infrared image library feature set and the reference feature set is calculated, and a cross-modal similarity matrix is ​​constructed. ;

[0036] S513: Projection network optimization features: Incorporating cross-modal similarity matrix Input a pre-trained pairwise matrix projection network Output a pairwise matrix with stronger identity differentiation:

[0037] If the input is a cross-modal similarity matrix constructed from a visible light image library... Output The overall structure constitutes the visible light optimized feature set. ;

[0038] If the input is a cross-modal similarity matrix constructed from an infrared image library... Output The overall structure constitutes an optimized infrared feature set. ;

[0039] S514: Feature Storage: Optimize the visible light feature set Or infrared optimized feature set The corresponding image's index information is associated and stored to form a feature database;

[0040] S520: Query Image Matching:

[0041] For the input query image, perform real-time matching. The specific steps are as follows:

[0042] S521: Backbone Network Extracting image features: Inputting the query image into the backbone network Output the query image features:

[0043] S522: Constructing a cross-modal similarity matrix :

[0044] For visible light queries: select the infrared feature set from the training phase. Using the reference feature set as a reference, the similarity between the visible light query image and the reference feature set is calculated to obtain the cross-modal similarity matrix of the query image. ;

[0045] For infrared queries: select the visible light feature set from the training phase. Using the infrared query image as a reference feature set, the similarity between the infrared query image and the reference feature set is calculated to obtain the cross-modal similarity matrix of the query image. ;

[0046] S523: Projection network optimization features: Optimize the cross-modal similarity matrix of the query image. Input pairwise matrix projection network Output paired matrices :

[0047] S524: Reversible Mapping Model Modality Transformation: Eliminates cross-modal differences by converting pairwise matrices of the query image into equivalent features of the target modality.

[0048] If the queried image is visible light and the image library is infrared: use the reversible mapping model. Forward transform, output Converted into a visible light-to-infrared feature set, and compared with the infrared optimized feature set. Modal consistency;

[0049] If the queried image is infrared and the image library is visible light: use the reversible mapping model. Inverse transformation, output Converted into infrared-to-visible light feature set, and compared with visible light optimized feature set. Modal consistency;

[0050] S530: Similarity Calculation and Matching Ranking: Calculate the cosine similarity between the query image feature set and all features in the library feature set. or Sort by cosine similarity value from high to low, and take the images corresponding to the top L features as the cross-modal identity re-identification results of the query image.

[0051] In the aforementioned unsupervised visible-infrared personnel re-identification method based on cyclic pairwise identity learning, during the training phase step S100, the cluster centers are dynamically maintained and initialized through a momentum update formula during training. , ,in, This represents the memory update rate, used to balance the historical information of cluster centers with the update weights of current features, ensuring the stability and adaptability of cluster centers.

[0052] The aforementioned unsupervised visible-infrared person re-identification method based on cyclic pairwise identity learning, in step S300 of the training phase, involves an invertible mapping model. Depend on It is composed of a series of INN reversible blocks, which achieve cross-modal feature distribution alignment through lossless reversible transformation, thus reducing modal differences while preserving core identity information.

[0053] The core concept of this invention is to refine the cross-modal identity correspondence by constraining the pairwise relationships of pedestrians, and to introduce cross-modal cyclic distribution alignment to align feature distributions in a non-invasive manner. The two work together to maximize the discriminative power of identity information while reducing modal differences. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the overall framework of the method of the present invention.

[0055] Figure 2 This is a schematic diagram of reversible mapping feature transformation. Detailed Implementation

[0056] The unsupervised visible light-infrared person re-identification method based on cyclic pairwise identity learning provided by this invention has the following process: Figure 1 As shown, the training phase specifically includes the following steps:

[0057] Step S100: Dual-modal feature extraction and initialization:

[0058] Input visible light image set and infrared image set ,in, Indicates the first Visible light image, Indicates the number of visible light images. Indicates the first 1 infrared image, Indicates the number of infrared images;

[0059] Using a pre-trained ResNet50 backbone network Features were extracted separately to obtain the visible light feature set. and infrared feature set ;in, Indicates the first Features of a visible light image Indicates the first Features of an infrared image;

[0060] Visible light feature set and infrared feature set The DBSCAN density clustering algorithm is applied to generate a set of cluster centers for each modality, denoted as . and ,in, Represents a visible light cluster set. Represents the first visible light Cluster centers, Represents an infrared cluster set. Indicating the infrared first There are 10 cluster centers, which are dynamically maintained and initialized during training using a momentum update formula: , ,in, This indicates the memory update rate.

[0061] Step S200: Calculate the identity loss by jointly analyzing cross-modal cluster centers and pedestrian features. The loss function is formalized as follows:

[0062] ,

[0063] ,

[0064] in, This indicates loss of visible light identity. Indicates loss of infrared identity. Indicates and Visible light cluster centers with the same pseudo-label Indicates and Infrared cluster centers with the same pseudo-labels Indicates transpose calculation. This indicates the number of cluster centers for visible light. This indicates the number of cluster centers in the infrared spectrum. It is a temperature parameter;

[0065] Ultimately, this will result in the loss of multimodal identity. The definition is as follows: .

[0066] S210: Iterative Identity Relationship Matching: To address the issue of potentially inaccurate cross-modal alignment of cluster centers, cross-modal cluster center matching is modeled as a bipartite graph matching problem and solved through iterative optimization. First, a matching matrix is ​​defined. Matching elements in the matrix Represents the first visible light Cluster centers With infrared Cluster centers Does it match (1 for yes, 0 for no)? Let represent a real number. Then solve the following optimization problem:

[0067] ,

[0068] ,

[0069] ,

[0070] ,

[0071] Among them, the cost matrix elements , The similarity is cosine. However, the cluster centers obtained from the visible and infrared modes may not be precisely aligned, and the matching matrix must be iteratively refined. An iterative matching strategy is adopted: it is assumed that the number of cluster centers in visible light is greater than the number of cluster centers in infrared light. After the initial matching, each infrared cluster center finds its best visible light cluster center match, while some visible light cluster centers remain unmatched. Subsequently, only the unmatched visible light cluster centers participate in the next round of matching, until all cluster centers have a corresponding relationship or no matching items are found. The final output is a visible light to infrared cross-modal matching dictionary. Infrared-to-visible cross-modal matching dictionary Used to derive visible light false tags and infrared fake tags .

[0072] S220: Pairwise Matrix Learning and Optimization: Directly using the similarity of the original features is not accurate enough. This invention designs a pairwise matrix projection network. (Composed of fully connected layers) to learn a better pairwise matrix .

[0073] 1) Calculate the cross-modal similarity matrix :

[0074] ,

[0075] in, Represents the cross-modal similarity matrix The elements in This represents the L2 norm of a vector.

[0076] 2) Through network mapping: This involves mapping the cross-modal similarity matrix... Projected into a more identity-distinguishing pairwise matrix middle, ,in, Represents a pairwise matrix, where each element in the pairwise matrix is ​​a pairwise matrix. express and Probability estimate of belonging to the same identity.

[0077] 3) Training a pairwise matrix projection network Defined as pairwise matrix loss for:

[0078] ,

[0079] ,

[0080] ,

[0081] in, Represents pairwise matrix loss. Indicates the loss due to false labels. and , This represents the relative order loss. The pairwise matrix loss aims to encourage cross-modal similarity matrix... With paired matrices The positive correlation between them, specifically, when and When pairwise matrices exhibit high similarity in their latent representations, Corresponding element in It should also reflect a high degree of similarity.

[0082] S230: Constructing a reliable multimodal relation matrix To suppress noise, pairwise matrices are used. Threshold filtering is performed to obtain a cleaner multimodal relation matrix. :

[0083] ,

[0084] in, For the preset threshold, Representing the multimodal relation matrix The elements in Indicates visible light pseudo-label The first in One element, Indicating infrared fake tags The first in Elements; based on a multimodal relation matrix It can calculate the visible light confidence vector of each feature and its cross-modal identity relationship. and infrared confidence vector :

[0085] ,

[0086] in, Represents the visible light confidence vector The first in One element, Infrared confidence vector The first in Each element.

[0087] S240: Calculate cross-modal identity alignment loss : Design a cross-modal identity alignment loss using the constructed reliable relationship information. This guides feature learning, enabling the model to focus more on high-confidence cross-modal positive sample pairs:

[0088] ,

[0089] ,

[0090] ,

[0091] in, Visible light to infrared cross-modal matching dictionary Found, and The corresponding cross-modal cluster centers, Infrared-to-visible cross-modal matching dictionary Found, and The corresponding cross-modal cluster centers, This indicates the visible light to infrared alignment loss. This indicates the alignment loss between infrared and visible light.

[0092] Step S300: Cross-modal cyclic distribution alignment based on an invertible mapping model: To address the problem of pseudo-label noise accumulation during training, this invention introduces an invertible mapping model. It aligns feature distributions in a non-invasive way, that is, it shortens the modal distance without losing identity information.

[0093] S310: Reversible transformation process: constructing including Invertible mapping model of invertible blocks of INN (Invertible Neural Network) For the first INN invertible blocks, whose inputs have Dimensions, dividing it into two segments Subsequently, two linear neural networks were used respectively. and deal with.

[0094] The formula for forward mapping (visible light → infrared) is defined as follows:

[0095] ,

[0096] ,

[0097] Then, combine and Input features that form the next invertible INN block .

[0098] The formula for the inverse mapping (infrared → visible light) process is defined as follows:

[0099] ,

[0100] ,

[0101] Then, combine and Input features that form the next invertible INN block .

[0102] The aforementioned forward and backward mappings are inverses of each other; therefore, the overall invertible mapping model can be represented. and its inverse function The definition is as follows:

[0103] , ,

[0104] in, Represents the visible light to infrared feature set. This represents the infrared-to-visible light feature set.

[0105] S320: Reversible Alignment Loss: The goal of training reversible INN blocks is to ensure that transformed features maintain their identity while bridging modal differences. Reversible Modal Alignment Loss The design is as follows:

[0106] ,

[0107] ,

[0108] ,

[0109] in, This represents the feature alignment loss. The multimodal relation matrix obtained in step S230 As weights, they drive cross-modal feature pairs with high matching probabilities to be close to each other in the feature space, achieving weighted distribution alignment; This represents a self-supervised consistency constraint, requiring that a feature should remain as unchanged as possible after going through a "modal transformation-return" loop. This effectively ensures that the core identity information is preserved during the transformation process of the INN invertible block. Visible light variable infrared feature set The first in One element, Infrared to Visible Light Feature Set The first in Each element.

[0110] Step S400: Joint Model Training and Optimization: Final Total Loss It is the weighted sum of the above losses:

[0111] ,

[0112] in, and To balance the hyperparameters.

[0113] The method of this invention involves three key components: backbone network Paired matrix projection networks and invertible mapping model Pairwise matrix projection networks The parameters within are determined by pairwise matrix loss. Independent training, while backbone network and invertible mapping model Then in total loss Joint optimization under constraints.

[0114] The training process employs an alternating strategy:

[0115] 1. Pairwise matrix projection network Using pairwise matrix loss Independent training.

[0116] 2. Backbone Network and invertible mapping model Total loss Joint training. The optimizer uses Adam with an initial learning rate of... The learning rate decreases by a factor of 10 every 20 training cycles, for a total of 50 training cycles.

[0117] The identification phase specifically includes the following steps:

[0118] Step S500: Identity Re-identification.

[0119] S510: Image Library Feature Processing (Offline Execution):

[0120] S511: For the target image database (visible light image database or infrared image database), pre-process and store features in batches, and then input the images from the target image database into the trained backbone network. Output feature set of the image library: If it is a visible light image library, then output If it is an infrared image library, then output... ;

[0121] S512: Constructing a cross-modal similarity matrix :

[0122] If it is a visible light image library: select the infrared feature set from the training phase. Using the visible light image library feature set as a reference feature set, the similarity between the reference feature set and the visible light image library feature set is calculated, and a cross-modal similarity matrix is ​​constructed. ;

[0123] If it is an infrared image library: select the visible light feature set from the training phase. As a reference feature set, the similarity between the infrared image library feature set and the reference feature set is calculated, and a cross-modal similarity matrix is ​​constructed. ;

[0124] S513: Projection network optimization features: Incorporating cross-modal similarity matrix Input a pre-trained pairwise matrix projection network Output a pairwise matrix with stronger identity differentiation:

[0125] If the input is a cross-modal similarity matrix constructed from a visible light image library... Output The overall structure constitutes the visible light optimized feature set. ;

[0126] If the input is a cross-modal similarity matrix constructed from an infrared image library... Output The overall structure constitutes an optimized infrared feature set. ;

[0127] S514: Feature Storage: Store the processed visible light optimized feature set. Or infrared optimized feature set The corresponding image index information is stored together to form a feature database.

[0128] S520: Image Match Query (Execute Online):

[0129] For the input query image (visible light query or infrared query), perform real-time matching. Specific steps:

[0130] S521: Backbone Network Feature extraction: Input the query image into the backbone network Output the query image features:

[0131] S522: Constructing a cross-modal similarity matrix :

[0132] For visible light queries: select the infrared feature set from the training phase. Using the reference feature set as a reference, the similarity between the visible light query image and the reference feature set is calculated to obtain the cross-modal similarity matrix of the query image. ;

[0133] For infrared queries: select the visible light feature set from the training phase. Using the infrared query image as a reference feature set, the similarity between the infrared query image and the reference feature set is calculated to obtain the cross-modal similarity matrix of the query image. ;

[0134] S523: Projection network optimization features: Optimize the cross-modal similarity matrix of the query image. Input pairwise matrix projection network Output paired matrices :

[0135] S524: Reversible Mapping Model Modality Transformation: Eliminates cross-modal differences by converting pairwise matrices of the query image into equivalent features of the target modality.

[0136] If the queried image is visible light and the image library is infrared: use the reversible mapping model. Forward transform, output Converted into a visible light-to-infrared feature set, and compared with the infrared optimized feature set. Modal consistency;

[0137] If the queried image is infrared and the image library is visible light: use the reversible mapping model. Inverse transformation, output Converted into infrared-to-visible light feature set, and compared with visible light optimized feature set. Modal consistency;

[0138] S530: Similarity Calculation and Match Ranking:

[0139] Calculate the cosine similarity between the query image feature set and all features in the library feature set: or (Traverse all features in the image library); sort by cosine similarity value from high to low, and take the images corresponding to the top L features as the cross-modal identity re-identification results of the query image.

[0140] Experimental results:

[0141] Experiments on the public datasets SYSU-MM01 and RegDB (see table below) demonstrate that the method of this invention significantly outperforms the state-of-the-art methods in all metrics under unsupervised settings, proving its effectiveness.

[0142]

[0143] Test modes: different cross-modal matching scenarios: SYSU-MM01: divided into "full search" (mixed matching of all scenarios) and "indoor search" (matching of indoor scenarios only); RegDB: divided into "visible light to infrared" (using visible light as the query image to search for infrared) and "infrared to visible light" (using infrared as the query image to search for visible light);

[0144] Metrics: Core evaluation criteria in the VI-ReID field:

[0145] Rank-1: The probability that the first result in the search results is the correct identity (the higher the better);

[0146] mAP: Average accuracy of all queries (the higher the better);

[0147] mINP: The average ranking of the most informative negative samples across all queries (the higher the better).

[0148] This method achieved high Rank-1, mAP, and mINP values ​​in different test modes on the SYSU-MM01 and RegDB datasets, indicating that this unsupervised cross-modal person re-identification method has good matching accuracy.

[0149] The main differences and beneficial effects of this invention compared to the prior art are as follows:

[0150] 1. Different technical concepts: Existing technologies mostly adopt a two-stage paradigm of "first clustering to generate pseudo-labels, then forcibly aligning features," which easily leads to error accumulation and loss of identity information. This invention proposes a framework that combines "relationship constraints" and "cyclic alignment," continuously injecting and utilizing fine-grained identity relationship information during the alignment optimization process to form a positive cycle.

[0151] 2. Pairwise relationship constraints for pedestrians were introduced: By iterative identity matching and learnable pairwise matrices, a more refined and reliable multimodal relationship matrix was generated than that of simple clustering pseudo-labels, providing high-quality guidance for subsequent alignment.

[0152] 3. A pioneering application of reversible mapping models to the VI-ReID task: Utilizing the lossless and reversible nature of INN reversible blocks for feature distribution alignment, this overcomes the shortcomings of traditional alignment methods that suffer from identity information loss due to irreversible transformations. The design of its feature alignment loss and self-supervised consistency constraints ensures that identity-discriminating features are preserved to the greatest extent possible while minimizing modal differences.

[0153] 4. Significantly superior recognition performance: As shown in the table above, the method of this invention significantly outperforms existing unsupervised methods in key indicators such as Rank-1 and average accuracy under multiple datasets and evaluation protocols. Some indicators even approach the performance of supervised methods, demonstrating its powerful ability to solve the problem of unsupervised cross-modal person re-identification.

Claims

1. An unsupervised visible light-infrared person re-identification method based on cyclic pairwise identity learning, characterized in that: It includes a training phase and a recognition phase. In the training phase, firstly, a backbone network is used. Multimodal features are extracted from pedestrian images, and then a cross-modal similarity matrix is ​​calculated based on these features. Through pairwise matrix projection networks Cross-modal similarity matrix Projected onto pairwise matrices Then, by merging them into pairs of matrices Using cross-modal identity matching information to calculate the multimodal relationship matrix Finally, the invertible mapping model is used. Align modalities and train the backbone network using loss during the training phase. Paired matrix projection networks and invertible mapping model ; In the recognition stage, the trained network and model are used to construct a library feature set for the target image library and a query image feature set for the input query image. The cosine similarity between the query image feature set and all features in the library feature set is calculated. The features are sorted from high to low according to the cosine similarity value, and the images corresponding to the top L features are taken as the cross-modal identity re-recognition results of the query image. Calculate the multimodal relation matrix The specific process is as follows: S220: Pairwise Matrix Learning and Optimization 1) Calculate the cross-modal similarity matrix : ,in, Represents the cross-modal similarity matrix The elements in The L2 norm of a vector; Indicates the first Features of a visible light image Indicates the first Features of an infrared image; 2) Cross-modal similarity matrix is ​​mapped through network mapping Projected into a more identity-distinguishing pairwise matrix middle, ,in, Represents a pairwise matrix, where each element in the pairwise matrix is ​​a pairwise matrix. express and Probability estimate of belonging to the same identity Indicates the first Visible light image, Indicates the first 10 infrared images; It is a pairwise matrix projection network; 3) Training a pairwise matrix projection network The pairwise matrix loss is defined as: , , ,in, Represents pairwise matrix loss. Indicates the loss due to false labels. and , Indicates relative order loss; Indicates the number of visible light images. Indicates the number of infrared images; Indicates a visible light pseudo-label. Indicates an infrared tamper-evident tag; S230: Constructing a reliable multimodal relation matrix To suppress noise, pairwise matrices are used. Threshold filtering is performed to obtain a cleaner multimodal relation matrix. : ,in, For the preset threshold, Representing the multimodal relation matrix The elements in Indicates visible light pseudo-label The first in One element, Indicating infrared fake tags The first in Elements; based on a multimodal relation matrix Calculate the visible light confidence vector of each feature and its cross-modal identity relationship. and infrared confidence vector : ,in, Represents the visible light confidence vector The first in One element, Infrared confidence vector The first in One element; S240: Calculate cross-modal identity alignment loss : , , ,in, Visible light to infrared cross-modal matching dictionary Found, and The corresponding cross-modal cluster centers, Infrared-to-visible cross-modal matching dictionary Found, and The corresponding cross-modal cluster centers, This indicates the visible light to infrared alignment loss. This indicates the alignment loss between infrared and visible light; Indicates the first Features of a visible light image Indicates the first Features of an infrared image; It's a temperature parameter. Represents the first visible light Cluster centers, Indicating the infrared first Cluster centers.

2. The unsupervised visible light-infrared personnel re-identification method based on cyclic pairwise identity learning according to claim 1, characterized in that: The training phase specifically includes the following steps: Step S100: Dual-modal feature extraction: Input visible light image set and infrared image set ,in, Indicates the first Visible light image, Indicates the number of visible light images. Indicates the first 1 infrared image, Indicates the number of infrared images; Use backbone network Multimodal features were extracted separately to obtain the visible light feature set. and infrared feature set ;in, Indicates the first Features of a visible light image Indicates the first Features of an infrared image; Visible light feature set and infrared feature set Clustering algorithms are applied separately to generate a set of cluster centers for each modality, denoted as . and ,in, Represents a visible light cluster set. Represents the first visible light Cluster centers, Represents an infrared cluster set. Indicating the infrared first Cluster centers; Step S200: Calculate the identity loss by jointly analyzing cross-modal cluster centers and pedestrian features. The loss function is formalized as follows: , ,in, This indicates loss of visible light identity. Indicates loss of infrared identity. Indicates and Visible light cluster centers with the same pseudo-label Indicates and Infrared cluster centers with the same pseudo-labels Indicates transpose calculation. This indicates the number of cluster centers for visible light. This indicates the number of cluster centers in the infrared spectrum. It is a temperature parameter; ultimately, it will affect the multimodal identity loss. The definition is as follows: ; S210: Iterative Identity Relationship Matching: First, define the matching matrix. Matching elements in the matrix Represents the first visible light Cluster centers With infrared Cluster centers Determine if a match is found, and then solve the following optimization problem: , , , ,in, Let be the cost matrix, and its elements , Cosine similarity; iteratively refine the matching matrix. The final output is a visible-to-infrared cross-modal matching dictionary. Infrared-to-visible cross-modal matching dictionary Used to derive visible light false tags and infrared fake tags ; S220: Pairwise Matrix Learning and Optimization 1) Calculate the cross-modal similarity matrix : ,in, Represents the cross-modal similarity matrix The elements in The L2 norm of a vector; 2) Cross-modal similarity matrix is ​​mapped through network mapping Projected into a more identity-distinguishing pairwise matrix middle, ,in, Represents a pairwise matrix, where each element in the pairwise matrix is ​​a pairwise matrix. express and Probability estimate of belonging to the same identity; It is a pairwise matrix projection network; 3) Training a pairwise matrix projection network The pairwise matrix loss is defined as: , , ,in, Represents pairwise matrix loss. Indicates the loss due to false labels. and , Indicates relative order loss; S230: Constructing a reliable multimodal relation matrix To suppress noise, pairwise matrices are used. Threshold filtering is performed to obtain a cleaner multimodal relation matrix. : ,in, For the preset threshold, Representing the multimodal relation matrix The elements in Indicates visible light pseudo-label The first in One element, Indicating infrared fake tags The first in Elements; based on a multimodal relation matrix Calculate the visible light confidence vector of each feature and its cross-modal identity relationship. and infrared confidence vector : ,in, Represents the visible light confidence vector The first in One element, Infrared confidence vector The first in One element; S240: Calculate cross-modal identity alignment loss : , , ,in, Visible light to infrared cross-modal matching dictionary Found, and The corresponding cross-modal cluster centers, Infrared-to-visible cross-modal matching dictionary Found, and The corresponding cross-modal cluster centers, This indicates the visible light to infrared alignment loss. This indicates the alignment loss between infrared and visible light; Step S300: Cross-modal cyclic distribution alignment based on an invertible mapping model, introducing an invertible mapping model. The reversible mapping model and its inverse function The definition is as follows: , ,in, Represents the visible light to infrared feature set. Represents the infrared-to-visible light feature set; Reversible mode alignment loss The design is as follows: by using feature alignment loss and self-supervised consistency constraints, identity information is ensured to remain intact during the transformation process. , Feature alignment loss : ,in, Visible light variable infrared feature set The first in One element, Infrared to Visible Light Feature Set The first in One element; Self-supervised consistency constraint : ; Step S400: Joint training and optimization of the model, final total loss It is the weighted sum of the above losses: ,in, and To balance the hyperparameters; An alternation strategy is used during the training phase: 1) Pairwise matrix projection network Using pairwise matrix loss Independent training; 2) Backbone Network and invertible mapping model Total loss Joint training.

3. The unsupervised visible light-infrared personnel re-identification method based on cyclic pairwise identity learning according to claim 2, characterized in that: The identification phase specifically includes the following steps: Step S500: Identity Re-identification: S510: Image Library Feature Processing: S511: For the target image database, pre-process and store features in batches, then input the images from the target image database into the trained backbone network. It also outputs the feature set of the image library: if it is a visible light image library, it outputs... If it is an infrared image library, then output... ; S512: Constructing a cross-modal similarity matrix : If it is a visible light image library: select the infrared feature set from the training phase. Using the visible light image library feature set as a reference feature set, the similarity between the reference feature set and the visible light image library feature set is calculated, and a cross-modal similarity matrix is ​​constructed. ; If it is an infrared image library: select the visible light feature set from the training phase. As a reference feature set, the similarity between the infrared image library feature set and the reference feature set is calculated, and a cross-modal similarity matrix is ​​constructed. ; S513: Projection network optimization features: Incorporating cross-modal similarity matrix Input a pre-trained pairwise matrix projection network Output a pairwise matrix with stronger identity differentiation: If the input is a cross-modal similarity matrix constructed from a visible light image library... Output The overall structure constitutes the visible light optimized feature set. ; If the input is a cross-modal similarity matrix constructed from an infrared image library... Output The overall structure constitutes an optimized infrared feature set. ; S514: Feature Storage: Optimize the visible light feature set Or infrared optimized feature set The corresponding image's index information is associated and stored to form a feature database; S520: Query Image Matching: For the input query image, perform real-time matching. The specific steps are as follows: S521: Backbone Network Extracting image features: Inputting the query image into the backbone network Output the query image features: S522: Constructing a cross-modal similarity matrix : For visible light queries: select the infrared feature set from the training phase. Using the reference feature set as a reference, the similarity between the visible light query image and the reference feature set is calculated to obtain the cross-modal similarity matrix of the query image. ; For infrared queries: select the visible light feature set from the training phase. Using the infrared query image as a reference feature set, the similarity between the infrared query image and the reference feature set is calculated to obtain the cross-modal similarity matrix of the query image. ; S523: Projection network optimization features: Optimize the cross-modal similarity matrix of the query image. Input pairwise matrix projection network Output paired matrices : S524: Reversible Mapping Model Modality Transformation: Eliminates cross-modal differences by converting pairwise matrices of the query image into equivalent features of the target modality. If the queried image is visible light and the image library is infrared: use the reversible mapping model. Forward transform, output Converted into a visible light-to-infrared feature set, and compared with the infrared optimized feature set. Modal consistency; If the queried image is infrared and the image library is visible light: use the reversible mapping model. Inverse transformation, output Converted into infrared-to-visible light feature set, and compared with visible light optimized feature set. Modal consistency; S530: Similarity Calculation and Matching Ranking: Calculate the cosine similarity between the query image feature set and all features in the library feature set. or Sort by cosine similarity value from high to low, and take the images corresponding to the top L features as the cross-modal identity re-identification results of the query image.

4. The unsupervised visible-infrared person re-identification method based on cyclic pairwise identity learning according to claim 2 or 3, characterized in that: In step S100 of the training phase, the cluster centers are dynamically maintained and initialized during training using the momentum update formula: , ,in, This indicates the memory update rate.

5. The unsupervised visible-infrared person re-identification method based on cyclic pairwise identity learning according to claim 2 or 3, characterized in that: In training phase step S300, the invertible mapping model Depend on It consists of a series of reversible INN blocks.

Images